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ABSTRACT 

This  thesis  is  divided  into  two  parts.  In  the  first  part,  the  problem  of  signal  detection  in 
fractional  Gaussian  noise  is  considered.  To  facilitate  the  study  of  this  problem,  several  results 
related  to  the  reproducing  kernel  Hilbert  space  of  fractional  Brownian  motion  are  presented. 
In  particular,  this  reproducing  kernel  Hilbert  space  is  characterized  completely,  and  an  alterna¬ 
tive  characterization  for  the  restriction  of  this  class  of  functions  to  a  compact  interval  [0,7]  is 
given.  Infinite-interval  whitening  filters  for  fractional  Brownian  motion  are  also  developed. 
Application  of  these  results  to  the  signal  detection  problem  yields  necessary  and  sufficient 
conditions  for  a  deterministic  or  stochastic  signal  to  produce  a  nonsingular  shift  when  embed¬ 
ded  in  additive  fractional  Gaussian  noise.  Also,  a  formula  for  the  likelihood  ratio  correspond¬ 
ing  to  any  deterministic  nonsingular  shift  is  developed.  Finally,  some  results  concerning 
detector  performance  in  the  presence  of  additive  fractional  Gaussian  noise  are  presented. 

In  the  second  part  of  the  thesis,  the  application  of  reproducing  kernel  Hilbert  space 
theory  to  the  problems  of  robust  detection  and  estimation  is  investigated.  It  is  shown  that  this 
'  approach  provides  a  general  and  unified  framework  in  which  to  analyze  the  problems  of  L2 
estimation,  matched  filtering,  and  quadratic  detection  in  the  presence  of  uncertainties  regarding 
the  second-order  structure  of  the  random  processes  involved.  Minimax  robust  solutions  to 
these  problems  are  characterized  completely,  and  some  results  concerning  existence  of  robust 


solutions  are  presented. 
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CHAPTER  1 
INTRODUCTION 

In  this  thesis,  we  consider  a  variety  of  problems  in  the  area  of  statistical  signal  process¬ 
ing.  The  work  is  divided  into  two  relatively  unrelated  parts,  the  common  factor  being  the 
application  of  reproducing  kernel  Hilbert  space  (RKHS)  theory  to  the  various  problems  con¬ 
sidered.  Since  each  part  of  the  thesis  is  essentially  self-contained,  each  comprising  a  separate 
chapter,  we  present  here  only  some  brief  introductory  remarks  and  an  overview  of  the  thesis. 
Each  of  the  two  major  chapters  includes  a  more  complete  introduction  to  the  material 
presented. 

In  the  first  part  of  the  thesis,  we  consider  the  problem  of  communication  in  the  presence 
of  noise  displaying  strong  long-term  dependence.  We  are  motivated  to  consider  this  problem 
by  the  prevalence  of  natural  phenomena  that  exhibit  behavior  indicative  of  such  dependence. 
Among  the  many  physical  processes  exhibiting  such  behavior  are  river  flows,  frequency 
fluctuations  in  oscillators,  current  fluctuations  in  semiconductors,  and  errors  on  communica¬ 
tions  channels.  Given  the  ubiquitous  nature  of  phenomena  displaying  long-term  dependence, 
it  seems  desirable  to  consider  the  problem  of  communication  in  the  presence  of  strongly 
dependent  noise.  We  consider  one  aspect  of  this  problem,  namely,  signal  detection  in  the 
presence  of  additive,  strongly  dependent  noise,  and  we  adopt  as  our  noise  model  the  class  of 
random  processes  known  as  fractional  Brownian  motions  and  the  associated  derivative 
processes  referred  to  as  fractional  Gaussian  noises. 

In  the  second  part  of  the  thesis,  we  consider  the  problems  of  robust  L 2  estimation, 
matched  filtering,  and  quadratic  detection  in  the  presence  of  uncertainty  regarding  the 
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statistical  structure  of  the  random  processes  involved.  In  recent  years,  the  game- theoretic 
minimax  approach  to  designing  robust  detection  and  estimation  procedures  has  been  studied 
by  many  authors.  We  also  employ  a  minimax  strategy,  but  we  formulate  and  analyze  the 
robust  detection  and  estimation  problems  in  an  RKHS  context.  This  approach  provides  a  gen¬ 
eral  and  unified  framework  in  which  to  analyze  these  problems  and  also  clearly  reveals  the 
underlying  similarities  of  the  problems. 

The  thesis  is  organized  as  follows.  In  Chapter  2,  we  give  a  brief  review  of  the  relevant 
RKHS  theory.  In  Chapter  3,  we  study  the  problem  of  signal  detection  in  fractional  Gaussian 
noise,  and  in  Chapter  4,  we  consider  the  robust  detection  and  estimation  problems.  In 
Chapter  5,  we  present  some  concluding  remarks. 
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CHAPTER  2 

REVIEW  OF  REPRODUCING  KERNEL  HILBERT  SPACE  RESULTS 

In  subsequent  chapters  of  this  thesis,  we  make  frequent  use  of  some  basic  results  from 
the  theory  of  reproducing  kernel  Hilbert  spaces.  We  give  below  a  brief  review  of  these 
results.  For  a  thorough  introduction  to  the  subject,  see,  for  example,  [74]. 

Let  I  be  any  index  set.  The  term  covariance  function  refers  to  any  symmetric, 
nonnegative-definite  function  K :  I2  — >  (C.  Associated  with  any  such  covariance  function, 
there  is  a  unique  Hilbert  space  H{K)  of  functions  defined  on  /  such  that,  for  all  t  e  /  and  all 
/  e  H{K\ 

K(  J)  6  H(K), 
and 

/(o-y./co^Vot). 

This  function  space  is  called  the  reproducing  kernel  Hilbert  space,  or  RKHS,  with  reproduc¬ 
ing  kernel  K,  and  it  is  well-known  (see  [1],  §1.2)  that  H{K)  consists  of  functions  /:!-><£ 
of  the  form 

/(•)  =  SfctfU),  {<►/}&  1  c <r,  £/, 

«=1 

and  their  limits  under  the  norm 

\\f\\u(K)  =  L  $i$jK(tj  ,tj  ). 

*.y=i 

Note  that  norm  convergence  in  H  (K)  implies  pointwise  convergence  on  /  since,  for  all  re/ 


and  all/,g  €  B(K), 
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\f(t)-g(t)\  =  \'4f  -jUWVm1 
^  \\f  -  £  II* (jc) 

Given  any  two  RKHSs  B(K0)  and  B(KX)  defined  on  the  index  set  /,  the  direct  product 
space  B(K0)  ®  B{Kx)  consists  of  functions  g :  I2  -*  <T  of  the  form 


g C,*)  =  2  yijK0(,ti)Kl(*,tj), 

‘.7=1 


and  their  limits  under  the  norm 


At  N 


II#  W(Ko)  =  X  H,"iipUK<^tkdi)Kx{tl,tj). 

‘.y=l  *./=l 

It  follows  (see  [1],  §1.8)  that  B  (Xq)  ®B(KX)  is  itself  an  RKHS  with  reproducing  kernel 


((f i.Xi);  (t 2t^7))  —  K^txdi)K\{X\,X'j), 


( f,-.Xi )  e  I2. 


If  X  4  {X (r );  t  €  /}  is  a  stochastic  process  with  covariance  function  Kx  and  mean 
function  m  €  B(KX),  then  B(X)  &  B(KX)  is  congruent  (i.e.,  isometrically  isomorphic)  to 
the  Hilbert  space  L2(X)  spanned  by  the  random  variables  (X(f),  re/)  (see  [48],  §2).  For 
any  g  e  H(X),  the  corresponding  element  in  L2(X)  is  usually  denoted  by  (X,  g)n(\)  a°d  *s 


characterized  by  the  property  that 


« 

)'  (X.  g)g(K)  » 

4 


g(t)  =  Cov«X  (r). 


It  follows  that,  for  all  t  e  /  and  g,h  e  B (K x). 


V  t  e  /. 


(2.1.1) 


(X,  Kx(-, <>)„X,  =  x«). 


=  \m>  *)#«)• 
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(2.1.2) 


and 


'I*.  *}< 


C°vj(X,  g)a(X)<  (x»  f)a(X) 


=  '\h,g)a0iy 


(2.1.3) 


For  our  purposes,  we  will  often  take  /  itself  to  be  a  Hilbert  space  H 0.  For  example,  H 0 
might  be  L\R ),  the  space  of  real-valued  functions  that  are  square-integrable  with  respect  to 
Lebesgue  measure  on  R .  In  this  case,  the  observations  are  regarded  as  a  generalized  random 
process;  that  is,  X  is  regarded  as  a  linear  operator  mapping  H0  into  the  space  of  square- 
integrable  random  variables  on  some  probability  space.  The  space  L2(X)  then  consists  of 
mean-square  limits  of  random  variables  of  the  form  X(f ),  /  s  #0,  and  a  covariance  function 
is  a  bilinear  form  K :  H  $  -»  R .  In  fact,  we  will  assume  that  all  covariance  functions  are 
bounded  bilinear  forms  so  that  they  are  generated  by  covariance  operators  on  H  0;  that  is,  for 
all/ ,g  e  H0, 

K(f,g)  =  E{X(f)X(g))=!S,Kg)ao, 

where  K:  H0  ->  H0  is  bounded,  linear,  self-adjoint,  and  positive  (see  [14],  Lemma  X.2.2). 
The  RKHS  corresponding  to  such  a  covariance  function  (denoted  by  H  (K)  as  well  as  H  ( K )) 
consists  of  bounded  linear  functionals  s :  H0  R  of  the  form 

*(•)=(•,  Kg)*o,  g  e  Hq, 

and  their  limits  under  the  norm 


I!5  !!«(*:)  -  ($  >  k£/»0- 


Equivalently  (see  [48],  §9),  H(K)  consists  of  functionals  s  of  the  form 

i(n=(Kiy.s;«.=(f,  / « »0. 

where  K*  is  the  square  root  of  the  operator  K  and  S  e  Hq  is  contained  in  the  closure  of  the 
range  of  K'A.  The  norm  of  s  in  H(K)  is  then 

11* II* (AT)  =(S*  S)*0- 

Of  course,  appealing  to  the  congruence  between  H0  and  its  dual  Hq,  we  could  just  as  well 
say  that  H (K)  consists  of  functions  s  e  H0  of  the  form  s  =  K*S,  where  S  is  contained  in 
the  closure  of  the  range  of  K%.  While  this  is  not  technically  correct  (since  H (K)  a  Hq),  it 
is  a  useful  way  of  thinking,  and  we  will  often  blur  the  distinction  between  functions  and  func¬ 
tionals  by  writing  statements  such  as 

s(f)=(f,s)aa,  f  e  Hq. 

The  following  two  results  are  useful  for  characterizing  RKHSs  and  finding  canonical 
representations  of  stochastic  processes.  Proofs  can  be  found,  for  example,  in  [20]. 

Theorem  2.1.1:  Let  /  be  any  index  set  and  let  K  be  a  covariance  function  defined  on  / . 
Suppose  there  exists  a  measure  space  (A J3  ,v)  and  a  set  of  functions  {/, ;  t  e  I)  C  L2(v) 
such  that 

K(s,t)  =  lfs(k)fMdv(k),  V  (S,t)  6  I2. 

A 

Then  H  (K)  consists  of  functions  of  the  form 


8(t)  =  Sft(X)g(k)dv(X), 


t  €  /, 
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where  §  e  span{/t;  tel).  Further,  for  all  g  ,h  e  H  (AT), 

v?  *  h }h(K )  -  fh  $•)£ v(X). 

A 

Theorem  2.1.2:  If  X  ^  {X  (r);  t  e  /}  is  a  zero-mean  stochastic  process  with  covariance 
function  Kx  such  that,  as  above, 

Kx(s  J)  =  jfs  (k)f&)dv(\),  V  (sa)  e  I\ 

A 

then  there  exists  an  orthogonal  process  Zx  on  A  with  associated  measure  v  such  that 

X(0  =  J/f(X)dZx(X),  el, 

A 

and  L2(X)  =  L2( Zx)  if  and  only  if  {/,;  f  €  / }  spans  L2(v).  Further,  for  all  g  e  H(X), 

(x-  $)»(X)  =  J^(X)dZx(X), 

A 

where  £  is  given  by  Theorem  2.1.1. 

The  final  two  results  of  this  section  relate  RKHS  theory  to  hypothesis  testing.  Theorem 

2.1.3  is  well-known,  and  excellent  discussions  of  it  can  be  found  in  [21]  and  [22].  Theorem 

2.1.4  is  due  to  Oodaira  [45]. 

Theorem  2.13:  Suppose  that  /  is  a  separable  metric  space  and  X  ^  {X(t);  t  e  I }  is 
Gaussian  with  continuous  covariance  function  /fx.  The  hypothesis  testing  problem: 

H0:  X  has  mean  zero 

Hji  X  has  mean  function  m. 


versus 
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is  nonsingular  if  and  only  if  m  e  H(X),  in  which  case,  the  likelihood  ratio  is  given  by 

L(X)  =  exp  |x,  m) m(K)  -  m)ff(X)] 

Theorem  2.1.4:  Let  /  be  a  separable  metric  space  and  let  X  4  {X(t );  t  <=  / }  be  a  Gaus¬ 
sian  process.  Consider  the  following  hypothesis  testing  problem: 

HqI  X  has  mean  zero  and  continuous  covariance  function  K0 

versus 

Hj.-  X  has  mean  function  and  continuous  covariance  function  K j. 

This  problem  is  nonsingular  if  and  only  if  the  following  three  conditions  are  satisfied. 

(i)  (Ki~  K0)  e  B  (K0)  ®  B  (Xq). 

(ii)  H(K0)  =  B(Kx)  (in  the  set  theoretic  sense),  or  equivalendy,  there  exist  constants 
0  <  c  <  C  <  <*»  such  that  (CX0  -  Kx)  and  (Kx  -  cK0)  are  both  nonnegative 
definite.  (This  is  often  abbreviated  cKq  <  Kx  c  CKq.) 


(iii)  mxe  H(K0). 
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CHAPTER  3 

SIGNAL  DETECTION  IN  FRACTIONAL  GAUSSIAN  NOISE 


3.1.  Introduction 

In  statistical  signal  processing  applications,  the  lack  of  independence  between  observa¬ 
tions  has  traditionally  been  handled  by  modeling  the  data  as  an  ARMA  process  with  relatively 
few  parameters.  Unfortunately,  there  are  many  physical  processes  that  exhibit  strong,  posi¬ 
tive,  long-term  correlations,  which  are  not  well-modeled  by  such  ARMA  processes.  Such 
long-term  dependence  is  very  often  observed,  for  example,  in  geophysical  data,  where  it  takes 
the  form  of  long  periods  of  high  or  low  values  (see  [18]  and  the  references  cited  therein). 
Similarly,  errors  on  communications  channels,  "...  appear  to  be  grouped  in  bursts,  which  are 
in  turn  grouped  in  bursts,  etc."  [33].  This  tendency  for  low  or  high  values  to  be  followed  by 
other  low  or  high  values  is  often  referred  to  as  the  Joseph  effect  It  is  indicative  of  a  process 
possessing  a  long  memory  and  is  perhaps  best  explained  in  terms  of  the  spectral  behavior  of 
such  a  process.  In  particular,  if  the  observed  process  is  stationary,  and  the  correlations 
between  observations  are  positive  and  fall  off  so  slowly  that  the  covariance  function  is  not 
integrable,  then  the  spectral  density  of  the  process  will  be  unbounded  at  the  origin.  The 
predominance  of  low-frequency  power  is  the  cause  of  the  Joseph  effect. 

An  important  class  of  physical  processes  exhibiting  strong  long-term  dependence  are 
those  with  I/f-type  spectral  behavior,  that  is,  spectral  densities  approximately  proportional  to 
/ 1-2W ,  where  /  represents  frequency  and  H  is  a  constant  in  the  range  l/i  <  H  <  3/2.  This  type 
of  spectral  behavior  is  observed  in  a  great  many  different  phenomena,  including,  for  example. 
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the  frequency  fluctuations  in  electrical  oscillators,  the  current  fluctuations  in  metal  films  and 
semiconductor  devices  [76],  and  the  loudness  fluctuations  in  speech  and  music  [72].  Many  of 
these  1/f-noises,  as  they  are  frequently  called,  appear  to  be  relatively  stationary  and  Gaussian; 
but  for  values  of  H  £  1,  it  is  not  clear  how  such  a  spectrum  should  be  interpreted  (since  no 
stationary,  L2  process  could  possess  such  a  spectrum).  Much  discussion  has  been  devoted  to 
this  problem  (see,  for  example,  [26],  [34],  and  [46]),  but  it  is  far  from  being  resolved.  For 
values  of  H  in  the  range  Vt  <  H  <  1,  however,  such  processes  can  be  modeled  as  generalized 
Gaussian  processes  or  as  the  (stationary)  increments  of  nonstationary  Gaussian  processes. 
This  is,  of  course,  analogous  to  the  relationship  between  white  Gaussian  noise  and  ordinary 
Brownian  motion.  In  this  thesis,  we  restrict  our  attention  to  values  of  H  in  this  range. 

An  early  attempt  at  modeling  Ilf -noise  was  made  by  Barnes  and  Allan  [5],  who  proposed 
modeling  the  phase  noise  in  oscillators  as  a  fractional  integral  of  white  noise.  (See  [44]  for  a 
discussion  of  fractional  integration  and  differentiation.)  The  corresponding  ///-type  frequency 
noise  would  then  be  modeled  by  the  increments  of  the  phase-noise  process.  The  particular 
fractional  integral  that  Barnes  and  Allan  proposed  was 

,a0- 

where  (B(r);  t  £  0}  is  a  Brownian  motion.  Unfortunately,  a  process  defined  in  this  way  does 
not  have  stationary  increments.  A  later  refinement  of  this  model  is  the  fractional  Brownian 
motion  (FBM)  process  introduced  by  Mandelbrot  and  Van  Ness  [36].  This  process,  which  is 
discussed  in  more  detail  in  the  sequel,  has  stationary  increments  that  exhibit  Ilf- type  spectral 
behavior.  In  fact,  in  a  certain  sense,  FBM  has  a  stationary  derivative,  called  fractional  Gaus¬ 
sian  noise  (FGN),  with  spectral  density  equal  to  /1-2W,  Vi<  H  <  1.  In  addition  to  being 
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stationary,  the  increments  of  FBM  are  also  self-similar,  that  is,  for  each  a  >  0  and  tQ  e  R , 

[BH(tQ+ax)  -  Bh (to);  x  e  R  }  =  aH  { Bw(x );  x  e  IR  }, 

where  is  an  FBM  and  denotes  equality  of  finite-dimensional  distributions.  Processes 
with  self-similar  and  stationary  increments  have  received  much  attention  in  recent  years  since 
they  possess  many  interesting  properties.  In  particular,  the  sample  paths  of  such  a  process  are 
fractals  as  defined  in  [35].  (For  discussions  of  the  various  properties  of  these  processes,  see, 
for  example,  [27],  [28],  [42],  [43],  [60],  [61],  [62],  [63],  [64],  and  [71].)  Because  of  its  sim¬ 
plicity  and  its  many  interesting  properties,  FBM  has  become  a  popular  statistical  model.  In 
addition  to  being  the  preeminent  model  for  long-term  dependence  ([8],  [18],  [19],  [29],  and 
[37]),  it  is  finding  increasing  application  as  a  model  for  image  texture  ([32],  [49],  and  [50]). 

In  this  chapter,  we  consider  the  problem  of  detecting  signals  in  the  presence  of  additive 
FGN.  We  begin,  in  Section  3.2,  with  a  discussion  of  the  properties  of  FBM,  including  a 
rigorous  treatment  of  FGN.  In  Section  3.3,  we  study  the  reproducing  kernel  Hilbert  space  of 
FBM.  In  Section  3.4,  we  discuss  the  problem  of  detecting  deterministic  signals  in  FGN,  and 
in  Section  3.5,  we  consider  nondeterministic  signals.  In  Section  3.6,  we  investigate  some 
aspects  of  detector  performance  on  FGN  channels.  Section  3.7  contains  some  concluding 
remarks. 

3.2.  Fractional  Brownian  Motion  and  Fractional  Gaussian  Noise 

The  class  of  fractional  Brownian  motions,  or  FBMs,  was  introduced  by  Mandelbrot  and 
Van  Ness  in  [36].  In  this  section  we  will  define  this  class  of  processes  and  discuss  some  of 
their  more  interesting  properties.  A  more  complete  development  is  given  in  [36]. 
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For  purposes  of  this  paper,  we  will  use  the  following  (slightly  specialized)  version  of  the 
definition  given  in  [36].  Motivation  for  this  definition  will  be  provided  below.  Let 
B  4  [B(0;  t  €  JR }  be  a  standard  Brownian  motion  and  let  xh<H  <1.  The  fractional 
Brownian  motion  process  Bw  4  t  e  R  }  is  defined  by 


1 

T(H+lA) 


JOr-cl"-* 


t  e  R. 
(3.2.1) 


t  o 

(Where,  for  t  <  0,  the  notation  "  J  ’*  should  be  interpreted  as  "  -J  ".)  Clearly,  B7/  is  a  zero- 

o  i 

mean  Gaussian  process  and  BH  (0)  =  0.  Notice  that  if  we  extend  our  definition  to  include 
H  =  16,  we  get 


t  e  R. 


In  this  sense,  FBM  can  be  regarded  as  a  generalization  of  the  familiar  Brownian  motion  pro¬ 
cess.  It  is  a  generalization  that  is  particularly  useful  for  applications,  as  we  shall  see  in  the 
sequel. 

The  covariance  function  of  Bw  is  given,  after  some  analysis,  by 

KBH(s,t)  =  ^j-[\s\w  +  \t\2H  ~  \t-s\2H),  s,t  e  R,  (3.2.2) 


where 


V„  k  VarfBffd)]  =  UlM&SSsm . 


(3.2.3) 


It  follows  from  this  covariance  structure  that  the  increments  of  Bw  are  stationary  and 
self-similar,  that  is,  for  each  a  >  0  and  1 0  e  JR , 
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(BffbffHi  x)  -  BH(t0 );  x  6  JR  }  =  aH  (Bw(t);  x  e  R  },  (3.2.4) 

where  denotes  equality  of  finite-dimensional  distributions.  This  property  implies  that  B„ 
is  statistically  the  same  on  all  time  scales.  This  also  implies  that  the  sample  paths  of  Bw  are 
fractals ,  as  defined  in  [35], 

From  (3.2.4)  it  follows  that,  for  any  8  >  0,  the  process 
B//.S  -  -g- {£//(/ +8)  -  t  e  R  }  is  a  zero-mean,  stationary  Gaussian  process  with 

covariance  function 


*«„,(*)  = 


VHbw'2 


+  1 


w 


-2 


I  x  I 
5 


2H 


I X I 


-  1 


2 H 


x  e  IR , 
(3.2.5) 


s  VhH(2H-1)\x\w~\ 


It!  »  8, 


and  power  spectral  density 

SB„,(“)  =  b HiWe~imdx,  0  *  0)  s  J? 

5  leal1-2",  0  <  1 0)5 1  <  1. 


(3.2.6) 


Equation  (3.2.5)  implies  that  the  process  Bw  5  is  mixing  and  ergodic  but  not  strongly  mixing 
(in  the  sense  of  Rosenblatt  [55]).  Hence,  the  increments  of  FBM  provide  a  simple  model  for 
processes  with  strong  long-term  dependence.  Moreover,  (3.2.6)  implies  that  the  increments  of 
FBM  provide  a  good  model  for  certain  processes  with  l#-type  spectral  behavior.  Although  it 
is  not  immediately  obvious,  (3.2.1)  represents  a  fairly  natural  way  to  define  a  process  with 
this  spectral  behavior.  We  present  in  the  following  paragraph  a  heuristic  development  of  the 
definition  of  FBM. 
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In  order  for  all  increments  to  be  stationary  and  exhibit  l//-type  spectral  behavior,  B/y 
should  have  a  zero-mean,  stationary  Gaussian  "derivative"  W#  with  covariance  function 

ATW „(*)  =  VhH(2H-1)\x\2M~2,  x  €  R, 
and  power  spectral  density 

Sw„(<o)  =  lo)l1_2W,  0*(oeR. 

That  is,  we  would  like  to  define  Bw  as 

t 

BH(t)&  jWH(x)dx,  teJR,  (3.2.7) 

o 

where  the  process  Ww  has  the  properties  described  above.  Following  Barnes  and  Allan  [5], 
we  define  WH  to  be  the  -order  fractional  integral  cf  white  noise,  but  since  white 

noise  is  defined  on  all  of  JR ,  we  need  to  define  our  fractional  integrals  accordingly.  That  is, 
we  let 

t 

— Ji -f  \t^t\"-*W(x)dx,  Vt  6  R, 

where  W  is  a  standard  white  noise  process.  Substituting  this  expression  into  (3.2.7)  yields 

1 1 

BH(t)  =  r(//1_iA) 1  x~5  1 H-*W(s)dsdx, 

which,  upon  changing  the  order  of  integration,  becomes 


IS 


*//(')  = 


1 

n h-u) 


0 

l 

*  ' 

t 

J 1 X—S  1  H~>hd'Z 

t 

W(s)ds  +  J 

p  p 

t 

Jlt-j  \H~3fldT 

W(s)ds 

0 

»  J 

0 

S 

•  - 

. 

i 

r  (H+'A) 


J(lr-slw-* 


Is  \H~l/l)W(s)ds  Jlr-s  \H~'AW{s)ds 

o 


Rewriting  the  white  noise  integrals  in  terms  of  a  standard  Brownian  motion  B,  we  get  (3.2.1). 

The  process  Ww  described  above  is  loosely  referred  to  as  fractional  Gaussian  noise,  or 
FGN.  Clearly,  no  such  process  actually  exists,  but  the  concept  of  FGN  and  its  relationship  to 
FBM  can  be  made  rigorous  by  defining  Ww  as  a  generalized  Gaussian  process;  that  is,  as  a 
linear  operator  acting  on  a  certain  subset  of  L\lR).  Then,  if  we  let  7[0,r]  be  the  indicator 
function  of  the  interval  [0,r],  we  get 

5//(O  =  Ww(/[0jl]),  V  t  e  R.  (3.2.8) 

(Note:  for  t  <  0,  we  let  /[0.x]  =  -/[f,o]-)  This  is,  of  course,  a  generalization  of  the  well-known 
relationship  between  Brownian  motion  and  white  noise,  and  one  can  define  FGN  in  much  the 
same  way  that  one  can  define  white  noise,  as  follows. 

Let  B  again  be  a  standard  Brownian  motion,  and  let  BH  be  an  FBM  derived  from  B  via 
(3.2.1).  It  is  well  known  that  there  is  an  orthogonal  increments  process  Z  4  (Z(co);  co  s  IR  ) 
with  mean  zero  and  variance  1/(2 rc)  such  that,  for  any  function  /  e  L2(IR)  with  Fourier 
transform  / ,  we  have 

J  /  (t)dB  (r)=J/  (-co)dZ  (co),  (3.2.9) 


and  L2(B)  =  L2( Z).  One  can  define  standard  white  noise  as  an  operator  W:  L2(/R)  — »  L2(B) 
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by 

W(f)A  jf(t)dB(t)  =  jf(-<o)dZ(G»,  f  e  L2(J?). 

Now,  let  us  define  a  new  process  Z^H  4  {ZWh(co);  to  e  R  }  by 

Z*H(a>)& jW^e  2 dZ (X),  (O  €  J?.  (3.2.10) 

o 

It  follows  (see,  for  example,  [2],  Chapter  2)  that  ZWw  is  a  zero-mean,  Gaussian  process  with 
orthogonal  increments  and  associated  measure  p.  given  by 

M-CA )  ^  'T“f lci)l1-2"dd),  A  Borel.  (3211) 

2 *A 

Let  Aw  c  L2{R)  be  defined  as 

A//  4  [g  e  L2(«):  |  6  Lfy)}.  (3.2.12) 

That  is.  A//  is  the  subset  of  functions  in  L\R)  whose  Fourier  transforms  are  square  integra- 
ble  with  respect  to  the  measure  |i.  We  define  the  fractional  Gaussian  noise  operator 
A//  — »  L2(ZW„)  by 

Ww(g)4  Jg(-m)dZWH(co),  g  e  Aw,  (3.2.13) 

and  we  will  use  the  notation 

J^(OWw(/)dr  4w„(g). 

It  follows  immediately  from  (3.2.13)  that,  for  all/,g  e  Aw, 
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El \f<,tWH{t)dt\g{s)WH{s)ds  f  =  El  j  f  (-co)dZ  Ww(co)  j  g  (-co)dZ  Ww(o)) 


~~^\f  ©5 1  «® 1 1_2//  ^  ©. 


E-  J/(OHfo(Odr  '-E*  J*(j)Wfc(j)<fr  '=0. 

.  J  v  - 

Also,  it  can  be  shown  (see  Appendix  A,  Lemmas  A.l  and  A.2)  that  L  l(f? )  n  L2(fl)  c  A// 
and,  for  any  / ,g  e  L  l(R )  n  L2(JR ), 

-7-  //(-oJifFoiTlcol^dcoa  VhH{2H-\)  J  J/(r)|Tr7lf-s \w'2dsdt. 

Hence,  for  any  f  g  e  L  l(R )  n  L2(f? ),  we  have 


Ej  |/(OW|f(fV*  Jg(5)^w(5)ds  f  =  ±  J/(-<B)rR57ltDl 


(3.2.14) 


=  VW//(2H-1)  J  J/(OfU71r-s  \  w~2dsdt, 

so  the  operator  W#  behaves  like  a  zero-mean,  stationary  Gaussian  process  with  covariance 
function  Ky/H  and  power  spectral  density  5  as  previously  defined. 


It  remains  to  be  shown  that  Bw  results  from  Ww  via  (3.2.8);  that  is. 
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S//(0  =  jWH(x)dx  -  Ww(7  [o./  j). 

o 


V  t  e  R. 


To  this  end,  we  recall  that 


where 


jWtf  (x)dT  =  | 7[0,](-a)dZWH(co) 


7  »  -J*gn(Q)XW-'/4)-r 

=  J/[0,]H*>)Ig>I*~//«  2dZ(o» 


_  r  g;av-l  -‘**n(®XW^ 

Eco 


=  f  —  ~--lcol^-tfg  . .  2dZ(co) 

-i.  *< 


=  J  /,  (-co)dZ  (co). 


/f(co)4  — - lo)l  /4~we  2. 

ICO 


(3.2.15) 


Since  /[0^]  e  L2(ji),  we  have  /,  6  L\E),  and  it  is  straightforward  to  verify  that  the  inverse 
Fourier  transform  of  f,  is  given  by 

/, W  =  [/(-.oWO'-tl''-’4  -  +  'mMIm'"'*].  *  «  *■ 

(3.2.16) 


It  follows  that 
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t  — 

$WH(x)dx  =  //t(-<o)dZ(o)) 

0  — 

<M 

=  J/,(t)dB(t) 

t 

lr-tl"-*-  Ixlw-%)d5(t)  +  j\t-x\H~'/ldB (x) 

o 

-  BH  ( t ), 

as  desired. 

We  close  this  section  with  a  brief  discussion  concerning  stochastic  integration  with 
respect  to  FBM. 

Let  be  an  FBM  and  let  {7C„  }~=1  be  a  sequence  of  partitions  of  a  compact  interval 
[a  ,b  ]  with  mesh  size  going  to  zero;  that  is, 

*»  =  Uo.  •  •  •  ,r„:  a  =  t0  <  <  •  •  •  <tn.l<tn=b], 

where 

lim  l7t„  I  =  Iim  max  (rf  -  ri—1)  =  0. 

n  — n— lS<£/» 

If  /  is  a  bounded  function  defined  on  [a  Jj  ],  we  can  define  the  integral  of  /  with  respect  to 
in  the  usual  L1- fashion,  by 
* 

f(x)dBH(x)  A  lim  2J<k)[BH(fi)  ~  (',-!>],  5,-  e 

a  «=1 

Of  course,  the  integral  is  only  well-defined  if  the  right-hand  limit  exists  and  is  the  same  for 
all  sequences  of  partitions  {7tn}“=1  with  lnn  I  — »  0.  We  can  also  define  "improper"  integrals 
as  limits  of  "proper"  ones;  for  example,  if  /  is  bounded  and  integrable  with  respect  to  Bw  on 
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all  compact  intervals  [-T  ,T],  then 

-  T 

jf(x)dBH(x)&  lim  jf(x)dB„(x), 

whenever  the  right-hand  limit  exists. 

Now,  let  Ww  be  the  FGN  corresponding  to  B //.  We  wish  to  establish  some  relation¬ 
ships  between  the  operator  and  integrals  with  respect  to  BH. 

Lemma  3-2.1:  If  [fn)~=  i  C L !(l? )  n  L\R)  is  a  sequence  of  functions  that  converges 
in  L Hi? )  and  L2{R )  to  a  function  / ,  then 

Um  jfn(x)WH(x)dx  =  jf(x)WH(x)dx.  (3.2.17) 


Proof:  Let  |i.  be  the  measure  defined  by  (3.2.11).  Recall  that 


\fn(x)WH{x)dx-  jf(x)WH(x)dx 


\  r 


=  E 


J  V 


J[f«(X)-/('C)lWw(X)dt 


o* 

=  ^J  l/«(©)  -/(<0)l2l0)l1-2Wd£0. 


Hence,  to  prove  (3.2.17),  it  is  sufficient  to  show  that  fn-+f  in  L2(p).  To  this  end,  notice 
that,  since  fn-*f  in  LX(R)  and  L2{R ),  fn  — » /  uniformly  and  in  L2(IR).  So,  for  any 
e  >  0, 
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-  i 

-i-  f  t/_  (CO)  —  f  (w)  1 2 1 0)  1 1_^ d 0)  £  -7- f  t /„(<!>)  —  /  (CO)  1 2 1  CD  1 1_2//  ^ CO 
2tc  2tc  -1 

•• 

+  “|  r/n(co)-/(co)l2do) 


^  -i-  f  elcul  1_2//do)  +  — e  (for  sufficiently  large  n) 

2it  J  2it 


-1 

2-tf 


2tc 


71(2-2//) 


e. 


Therefore,  /„  — »  /  inLz(|x)  and  (3.2.17)  follows.  ■ 

Lemma  3.2.2:  If  /  is  a  bounded  function,  continuous  almost  everywhere  on  a  compact 
interval  then 


J/(t)dew(t)  =  |/(t)^w(t)dt. 


Proof:  Let  {7Cn  }(J1I  be  a  sequence  of  partitions  of  [a  J>]  with  l7t„  I  -»  0.  Then 


jf  (x)dBH  (x)  =  lim  X/(^)[fiw(r,)-Bw(r,_i)] 

a  »-*-  i=l 


=  Urn  £/<5i>  J 

n~*“  |=1  — 


where 


=  lim  J 


S/  ^[ff-iAl^ 


i=i 


WH(x)dx 


=  lim  J/„(x)^//(x)dx,. 

/i * 


/.(X)  4  2/fe)/|„.lJ,)(x). 


t=l 


Now,  since  /  is  continuous  almost  everywhere  on  [a,b],  /„  — ►  fl[a,b)  almost  everywhere  on 
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JR,  and,  since /  is  bounded,  the  dominated  convergence  theorem  implies  that  /„  ->  fl[a<b\  'n 
L l(R )  and  L2(JR ).  Hence, 

b 

jf(x)dBH(x)  =  lim  jfn(x)WH(x)dx 

a  • 

mm 

-  !  (by  Lemma  3.2.1) 

b 

=  jf(x)WH(x)dx.  U 

a 

Lemma  3.2-3 :  If  /  e  L1^?)  n  L\R)  is  bounded  and  almost  everywhere  continuous  on 
R,  then 

J/(x)dBw(x)  =  J/(x)Ww(x)dx. 

Proof:  By  definition. 


-  N 

j  f  (x)dB„  (x)  =  lim  Jf  (x)dBH  (x) 

—  N~*~  -N 


N 

=  Jim.  J/(x)VP//(x)dx 


=  ^lim  jfN(x)WH(x)dx, 


(by  Lemma  3.2.2) 


where 


fN(l)^f(l)I[-N#](l)- 


Clearly,  fN  /  in  Ll(R )  and  L2(J?),  so 
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jf(x)dBH( t)*  lim  jfN(x)WH(x)dx 

=  Jf  (x )WH  (x)d  x  (by  Lemma  3.2. 1).  ■ 

These  lemmas  are,  of  course,  not  exhaustive,  but  they  illustrate  that,  for  sufficiently 
well-behaved  functions  / ,  the  random  variable  W#  (f )  is  equivalent  to  the  integral  of  /  with 
respect  to  B# . 

3.3.  RKHS  Results  for  Fractional  Brownian  Motion 

In  this  section,  we  characterize  the  RKHSs  for  the  class  of  FBMs  and  present  some 
related  results.  In  particular,  we  develop  several  results  concerning  the  restriction  of  an  FBM 
to  a  compact  index  set  [0,r],  which  will  be  directly  applicable  to  the  problem  of  detecting 
signals  in  FGN.  We  begin  by  considering  the  unrestricted  FBM  process,  in  which  the  index 
set  is  R .  For  this  case,  we  have  the  following  result 

Theorem  3-3-1:  Let  Bw  be  an  FBM  and  let  p.  be  the  measure  defined  by  (3.2.11).  Then 
H(Bh)  consists  of  functions  of  the  form 

g(t)  =  T“  f  teR,  (3.3.1) 

2?t  id) 

where  g  e  L2(p).  Further,  for  all  g  ,h  e  H  (BH ), 

(g,  h)m  b„)  =  “/  McollWlcol^dct).  (3.3.2) 

Proof:  Let  W#  be  the  FGN  corresponding  to  BH  and  let  Zy/H  be  the  orthogonal  incre¬ 
ments  process  related  to  Ww  by  (3.2.13).  Then 
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KBH(s,t)  =  E{BH(s)BH(t)} 


=  E 


jWH(x)dxjWH(x)dx 


=  E 


j  f [O4  Ww  (0>)  J  /  [04  ](-«>)dZ  Ww  (“) 


-r-7 


2re  _L  i  0)  i  © 


I  co  I  d(o. 


It  can  be  shown  (sec  Appendix  A,  Lemma  A.3)  that  the  set  of  functions  { — — — ;  t  e  1R  } 
spans  L2(jx),  so  (3.3.1)  and  (3.3.2)  follow  by  applying  Theorem  2.1.1.  ■ 

Theorem  3.3.1  gives  a  frequency-domain  characterization  of  H(BW).  The  following 
corollary  gives  the  corresponding  time-domain  characterization. 

Corollary  33.2:  A  function  g  €  H{Bh)  if  and  only  if  there  exists  g*  e  L2(R)  such 
that 


,  ' ' 

g (t)  =  ;  -  -J / (s -x)H~^gHx)dxds , 


(3.3.3) 


Further,  for  all  g  ,h  e  B  (BH ), 


*)m( b„)=  j  h*(s)g*(s)ds. 


(3.3.4) 
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Proof:  By  Theorem  3.3.1,  g  e  B(BH)  if  and  only  if  there  exists  £  e  L2(j. t)  such  that, 
for  all  t  €  R, 


g(t)  =  ~  I  ■~-^IWlG>l1-2//d£0 

2it  i(0 

om 

■il 


where 


and 


l_g-»c*  isgn(a>)(//-V4)y 


10)1  ""‘e 


ia_«  <  sgn(<»)(//  -Vi)— 

£  (— 0))  I  ci)  I /W/  e  2 


dco 


=  ^  //f(o))|Wdco, 


,-JCOf 


/t (co)  £ 


ijgn(coXW-1/4)-| 


id) 


(cf.  (3.2.15)), 


£*(©)  ^|(-o))l<al*-//e  2. 

Qearly,  /,,£*  e  L2(R ),  so  let  g*  denote  the  inverse  Fourier  transform  of  g*  and  recall  from 
(3.2.16)  that  the  inverse  Fourier  transform  of  /,  is  given  by 


fM  =  ftHlm  -  ui"-*)  +  /[0j)WU-tl«-«], 


T  e  R. 


Hence, 
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g(t)  *  *—  J M<o)gm«x»d(o 

=  jft(x)g*(x)dx 

i  0  ‘ 

~1\H+W)  J(,r-T|W-V4~  ltlw-V4)J*(tTdx  +  Jlr-*lw_%g*CcjdT 

.  '  •»  _ 

=  rm  iaJ J is-if-^g^dxds, 

1  (ti—n)  o_ 


which  proves  (3.3.3). 

To  prove  (3.3.4),  we  proceed  in  a  similar  fashion.  Let  g  ,h  e  U  (By )  and  let 
g  Ji  €  L2(h)  be  given  by  Theorem  3.3.1.  Then  (3.3.2)  yields 

•• 

ig.  h)a(fiH)  ~  S  Mco)g (to)  leal  1-2Wd<D 

sgn  (©K//-V4)-2-  i  sgn(a))(//-V4)i 

2  (-<i>)  1  to  I  Vi“"  e  2  Jdto 

=  ~  f  h*(co)g*(£o)dci)  (h*,g*  defined  as  before) 

2tc 

=  J/»*(t)gWdt, 


=  ~J  [h(-a>)lG>r>*-we< 


which  proves  (3.3.4).  ■ 

Remark:  Notice  that  (3.3.3)  implies  that  any  function  g  €  H(Bh)  is  differentiable 


almost  everywhere  in  K  and  that  the  derivative  g'  is  the  (H-x/i)th- order  fractional  integral  of 
g*.  That  is,  for  almost  all  t  €  R , 
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/ 

g'(0  =  Y(H~A)  I  t.  (3.3.5) 

Functions  of  the  form  (3.3.5)  are  variants  of  the  so-called  Riesz  potentials,  which  have  been 
widely  studied  (see,  for  example,  [59]).  This  representation  can  be  exploited  to  give  various 
conditions  necessary  for  g  6  H(BW).  For  example,  it  follows  from  (3.3.5)  and  a  general 

i 

form  of  Young’s  inequality  (see  [15],  page  232)  that  g'  €  L  l~H (#? ).  Similarly,  from  (3.3.1) 
we  can  get  a  simple  sufficient  condition  for  g  e  H  (Bw ).  Letting  g '  be  the  Fourier  transform 
of  g\  it  follows  from  (3.3.1)  that  g  e  H(BW)  if  g'  e  Ll(R)  and 

oo 

j  l£'(o))l2lcol2W~1da>  <  <*>. 


The  following  theorem  gives  an  infinite  interval  whitening  filter  for  FBM. 

Theorem  3-3-3:  Let  B  be  a  standard  Brownian  motion  and  let  Bw  be  an  FBM  derived 
from  B  via  (3.2.1).  Then 


5(0  = 


1 


TQht-H) 


o  t 

J {\t~x\l/t~H  -  I x I X/*~H )dBH (x)  +  J I r-x I 'h~H dBH (x) 
—  o 


(3.3.6) 


Proof:  Again,  let  Ww  be  the  FGN  corresponding  to  Bw,  ZWw  the  associated  orthogonal 
increments  process,  and  p.  the  measure  given  by  (3.2.11).  Also,  let  Z  be  the  orthogonal  incre¬ 
ments  process  related  to  B  by  (3.2.9).  Then,  for  all  t  €  R , 
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B(t)=jdB(t) 

o 

=  j  l[0j](-<o)dZ  (to) 

=  /  W- o>)l<ol"V^“X"^ 


where 


dZw,(tt)  (by  inverting  (3.2.10)) 


r  e““-l  «  u.  *'*8n(®X//-V4)-r 

-  f - - i-lo)l//-V4e  2 


=  /— 

t<a 


=  j  f  t  (-<ri)dZ  yy  ^  (CO) , 


l-e-,03f  „  w  -*  sgn(a>)(//  -V4)— ■ 

/,(co)  4  — - Icol*-**  2 


1  CD 


Clearly,  ft  e  L2(ja),  and  it  is  straightforward  to  show  that  ft  e  L2{R )  with  inverse  Fourier 
transfonn /,  €  Ll(R)  n  L2(R)  given  by 


/,( T)  =  [/(-..0)(t)(l»-tl'W'  -  ITI14-")  +/I0J)(T)l(-tl‘'W'], 


x  e  R. 


If  we  let 


/atW  i 


1  -/ 


1  W  iV 


1  N  N ' 


I  [-NJ^](^)ft  (x)> 


X  6  R  , 


then  fN  -*  f,  in  Ll(R)  and  L2(R),  and 
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B(o=  IM-m zw„(o)) 

am 

=  jft(x)WH(x)dx 

am 

=  lim  f  fN(x)WH(x)dx  (by  lemma  3.2.1) 

N—*»» 

=  lim  f  fN(x)dBH(x)  (by  lemma  3.2.3) 

N-*~  Zm, 

=  jft(x)dBH(x) 

t 

t-x -  Ixl1^-//)^e//(x)  +  Jlr-cl^dB/zd)  .  ■ 

o 

We  now  consider  the  case  of  an  FBM  restricted  to  a  compact  index  set  [0,7].  Let  B„ 
be  an  FBM  and  let  Bwir  4  [BH\T(t)\  t  €  [0,7]}  represent  the  restriction  of  BH  to  [0,7].  It 
is  well-known  that  H(BH\T)  must  consist  entirely  of  functions  in  H{ Bw)  restricted  to  the 
interval  [0,7].  Unfortunately,  this  is  not  a  very  practical  characterization,  since  it  is  typically 
rather  difficult  to  determine  from  its  values  on  [0,7]  whether  a  particular  function  can  be 
extended  to  all  of  J?  in  such  a  way  that  the  extension  is  a  member  of  H  (Bw ).  Fortunately, 
there  is  a  more  useful  description  of  H  (BH  ^),  as  the  following  theorem  shows. 

Theorem  3.3.4:  B (BH  ( T)  consists  of  functions  of  the  form 

*«>  =  p  ,,-f  j^z-sf-^dz  glsTs'^ds,  t  €  [0,T],  (3.3.7) 

1  (n  -n)  q  * 

where  £  e  L2([0,7]).  Any  such  function  £  has  a  derivative  g'  almost  everywhere  in  [0,T], 
and,  for  almost  all  t  s  [0,7], 
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ut)  =  t 


=  tH-H— 
dt 


i-TTT-  j(t -S  j^H S^Hg  '(S  )ds 


r(3ti-H)Jo 


Further,  for  any  gjt  e  H  (BH  lT), 


*  h)a(BHIT)  -  \h{x)gdt)dx. 
o 


Proof:  Since  Bw ,7-  is  the  restriction  of  Bw  to  [0,7],  we  have,  for  all  s,t  e  [0,7], 


K B„ ,T(S  j)  -  K b*  ^  ) 


=  ~  J  / (o^ j(— <o)7 £0^j(-<o)  Icol 1  wd(o 


TT 


=  VHH(2H-l)jjl[0j](a)Im(x)\x-c\w~2dxda  (by  (3.2.14)) 

00 

=  -rg-2W)c°gWjJ|t^|2H-2rf^0  (by  (3.2.3)). 

71  00 

The  function  Ix-ol2^-2  can  be  decomposed  as  (see  [75],  Equation  (15)) 


3  min(o,T) 

t-ffl2"-2* - n±H) - (<„)«-*  f 


min(0,T) 

I 


(3.3.8) 


(3.3.9) 


Hence, 
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KB„,r(S't)  ~ 


-r(2-2H)cosizH 

K 


St 


jj\z-<5\w~2dzda 

00 


St 


i 


T(H-lA) 


min(o.T) 

j  u  l~w  ( a-u  )H  ~3f\z-u  )H  ~^du 


dxda 


2  min (sj) 

j  u'-w  UoK-^o-uf^da 


du 


=  J/,(a)/,W“, 

o 


where 


(3.3.10) 


It  can  be  shown  (see  Appendix  A,  Lemma  A.4)  that  the  functions  {/,;  t  €  [0,7])  span 
L2([0,T]),  so  (3.3.7)  and  (3.3.9)  follow  from  Theorem  2.1.1.  To  establish  (3.3.8),  we  choose 
an  arbitrary  function  g  e  B(BH  {T)  and  use  (3.3.7)  to  write 


i 


gTsJs'^ds 


- 1 - 

n H-'h)  l 


j(z-s)H-hs^H  f(T)ds 


dz. 


It  follows  that,  for  almost  all  t  e  [0,7], 


gX  0  = 


H-Vi 


nH-H)  i 


j(r-5  l/*~H  g  (s  )ds. 


Hence, 
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n«-w)j 


ds 


- - - - {u^gju) 

r(3/2-H)r(H-'A)  l  *K  } 


J(r  -s  )1/4-w  (s  -u  )H  ~^ds 


du 


=  ju 1A-W  g  (u  )du. 


Therefore,  for  almost  all  t  €  [0,7], 


.h-'aA. 

dt 


“ -  J(*  “5  5  g  \S  )dS 


m-H)i 


=  tH  ~'A—  ju 1/4-W I  (m  Vm 
dti 


*1(0, 


which  proves  (3.3.8).  ■ 

We  can  also  find  a  canonical  representation  for  B#  (r,  different  from  (3.2.1),  which  leads 
straightforwardly  to  a  finite-interval  whitening  filter,  as  follows. 

Theorem  3.3.5:  Let  {/,;  t  e  [0,7]}  be  as  defined  in  (3.3.10).  There  exists  a  process 
B T  4  [BT(t);  t  e  [0,7]},  which  is  a  standard  Brownian  motion  on  [0,7],  such  that,  for  all 
t  e  [0,7], 


Bff  1 7* (f )  =  j//(w)dB7( m) 


Jt/,_v*(t-m  )H~^dx 


dBT(u ), 


(3.3.11) 


and 
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1  1  1 

bT (!)  =  -r~.3.  •  „ .  W^dx\<?-u )^H u'Ar~HdBH  ,T(«),  (3.3.12) 

i(  hr-H  )  jj  q 

where  the  integral  in  (3.3.12)  is  to  be  interpreted  as  an  integral  with  respect  to  the  orthogonal 
increments  process  Z7  &  {ZT(t);  t  e  [0,7]}  given  by 

Zj( t)  =  dBtf  iy(x),  r  €  [0,7],  (3.3.13) 

Proof:  From  the  proof  of  Theorem  3.3.4,  we  know  that 

T 

=  S/s(u)f,(u)du,  v  s,t  e  [0,7]. 

0 

It  follows  from  Theorem  2.1.2  that  there  exists  a  zero-mean,  orthogonal  increments  process 
Br  4  lBT(t);  t  e  [0,7]}  with  associated  measure  Lebesgue  measure  such  that 

T 

BH  I t(0  =  jft(u)dBT(u )• 

0 

Since  the  functions  { ft ;  t  e  [0,7]}  span  L2([0,7]),  Theorem  2.1.2  also  guarantees  that 
LHBhit)  -  L>2(Rt),  which  implies  that  Bj  is  Gaussian.  Hence,  Bp,  is  a  zero-mean,  Gaus¬ 
sian,  orthogonal  increments  process  on  [0,7]  with  associated  measure  Lebesgue  measure.  It 
follows  that  Bj  is  a  standard  Brownian  motion  on  [0,7],  which  proves  (3.3.1 1). 

To  prove  (3.3.12),  we  define  a  new  orthogonal  increments  process 
Z T  A  [Zrity,  t  e  [0,7]}  by 

t 

ZT{t)  ^  jxA~H dBT(x),  t  6  [0,7].  (3.3.14) 

0 


This  definition  implies  that 
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Br(0  =  J^dZj-Cc),  v  t  e  [0 ,T],  (3.3.15) 

o 

so  (3.3.12)  will  be  proven  if  we  can  establish  (3.3.13);  that  is,  if  we  can  show  that 

Zr«)  =  irW.  v  <  s  [OX], 

To  this  end,  let  {it„ }  ~=1  be  a  sequence  of  partitions  of  the  interval  [0,7]  with  l7tn  I  — »  0. 
Then,  for  all  t  6  [0,7], 

it(t)  ,  to  ,rW 

=  So  ,SL  rc^)  £'  l“-*l<i'  >'W' ir«i)-8»  irft-i)l 

T 

=  fim  Jim  )(!-$,  («)-/,,.,(■<  )]dBr(u  )  (by  (3.3.11)) 

=  ]ta  Jim  - i/[u^1&)(f-?i)'W'4i'4'''t/'t(«)-/„.,(u)l  ’dBT(u) 

0  i-1  d 

T 

=  So  nVtf)J'.Si 

0  >  i—1  „ 

■SnC3>f 

•  a 

Now,  it  is  straightforward  to  show  that 


whence  it  follows  that 
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=  J/{0,](u)u,/4_//<iBT(u) 
o 


o 


=  Zr(/)  (by  (3.3.14)). 


This  establishes  (3.3.13)  and  proves  the  theorem.  ■ 

The  previous  two  theorems  admit  the  following  useful  corollary. 

Corollary  3.3.6:  Given  any  g  e  H{ BW)T),  the  corresponding  element  in  l\ Bw,t)  is 
given  by 

r _ ■ 

(3.3.16) 

Proof:  Let  Br  and  2y  be  as  in  the  proof  of  Theorem  3.3.5.  It  follows  from  Theorem 


2.1.2  that 


(SfflT*  #)»( B„,r)  -  J#(f)<®r(f)» 
0 


where  g  €  L2([0,T])  is  given  by  Theorem  3.3.4.  Hence, 
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(bh  it*  £/*(Bwir)  ~  J#  ({ Writ ) 
o 


dBr(0  (by  (3.3.8)) 


=  — r^— fr2""1  ~-f(t-x)x*-HTA-HgX't)dx  dZr(t)  (by  (3.3.15)) 

1  ( rl—n  )  q  «  q 


1 

» 

IT 

4  [ 

a/  i 


(by  (3.3.13)). 
o 


which  proves  (3.3.16).  ■ 

Remark:  The  RKHS  B(BH  iT)  was  also  studied  in  an  early  paper  by  Molcan  and  Golo- 


l 

sov  [38],  in  which  it  is  claimed  that  /  e  B (Bw  ,r)  if  and  only  if  /'  e  L  l~H  ([0 ,T]).  This  is 
certainly  a  necessary  condition  (as  discussed  in  the  remark  following  Corollary  3.3.2)  but  does 
not  seem  to  be  sufficient  Molcan  and  Golosov  also  give  a  canonical  representation  for  Bw  lT, 
which  is  essentially  equivalent  to  (3.3.11)  and  (3.3.12);  however,  the  result  is  presented 
without  proof. 


3.4.  Detection  of  Deterministic  Signals  in  FGN 

In  this  section,  we  consider  the  problem  of  detecting  a  deterministic  signal  in  the  pres¬ 
ence  of  additive  FGN  on  a  compact  observation  interval  [0,J].  This  corresponds  to  the  fol¬ 
lowing  hypothesis  testing  problem: 
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Hq:  dY(t )  =  dBH  | T(r),  r  s  [0,7] 

versus  (3.4.1) 

Ht:  dT(0  =  s(t)dt  +  dBH  IT(r),  t  e  [0,7], 

where  Y  4  {7(0;  r  €  [0,7']>  is  the  observed  process,  Bwir  is  an  FBM  restricted  to  [0,7], 
and  s  is  a  real-valued  function  defined  on  [0,7].  Problem  (3.4.1)  can  be  stated  more 
rigorously  as 

Hq:  Y(t)=BHlT(t),  t  6  [0,7] 

versus  (3.4.2) 

Hi:  Y(t)  =m(r)  +  BHiT(t),  t  e  [0,7], 


where 


i 

m(t )  4  js(x)dz. 

o 


(3.4.3) 


We  would  like  to  know 

(i)  under  what  conditions  on  the  signal  m  is  Problem  (3.4.2)  nonsingular,  and 

(ii)  in  the  event  that  (3.4.2)  is  nonsingular,  what  is  the  formula  for  the  likelihood  ratio? 
We  can  make  use  of  the  results  from  the  preceding  section  to  answer  (i)  and  (ii),  as  follows. 

Theorem  3.4.1:  Problem  (3.4.2)  is  nonsingular  if  and  only  if 


m(t)  = 


- - J 


r (ff-fc) 


J'cH~'A(x-<y)H~3fldx 

m  (o)ax/*~H  d  a. 

v  t  €  [0,7],  (3.4.4) 

a 
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where  m  e  L2([0,T]),  and,  for  almost  all  t  €  [0,T], 


m(')  =  r»-«4 

dt 


n3/2-//)J0 


(3.4.5) 


Proof:  Since  m  is  real- valued,  Y  is  Gaussian  with  covariance  function  Ky  =  Krhit 
under  both  hypotheses,  so  Problem  (3.4.2)  reduces  to 


versus 


Hq:  Y  has  mean  zero 


Hjt  Y  has  mean  function  m. 


Hence,  by  Theorem  2.1.3,  (3.4.2)  is  nonsingular  if  and  only  if  m  e  H(Y).  Since 
Ky  =  ATBhit.,  we  have  H  (Y)  =  H(Bh  ,r),  so  (3.4.4)  and  (3.4.5)  follow  immediately  from 
Theorem  4.4.  ■ 

Theorem  3.4 J:  If  m  satisfies  (3.4.4)  so  that  Problem  (3.4.2)  is  nonsingular,  then  the 
likelihood  ratio  is  given  by 

L(Y)  =  exp  |y,  m  )a  ^  -  xh  [m ,  m)B ^  j ,  (3.4.6) 


where 


\m>  m/»(Y)  = 


■  • 

2  T 

,  t 

1 

f  t2H-\ 

Xs4-"  m'Wdx 

dt  o 

>  - 

m-H) 

V 

0 

dt. 


(3.4.7) 


and 
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(Y*  m)aon  - 


1 


row/) 


2  T 


,2W-l 


-7-  f(*  -x)1/i-W  m  \x)dx 

ut  a 


dj(f-x) 

o 


'A-H'Ar-H 


dY  (X). 
(3.4.8) 


Proof:  If  m  satisfies  (3.4.4),  then  m  e  H  (Y)  =  H  (B#  (y),  and  (3.4.6)  and  (3.4.7)  follow 
immediately  from  Theorems  2.1.3  and  3.3.4,  respectively.  To  prove  (3.4.8),  we  will  show 
that  for  any  function  g  e  H( Y),  (Y,  g)g<y)  ^  given  by 

(Y»  ^)ar(Y>  =  4Kg) 


■ 

2  r 

,  < 

4 

1 

rO/2-//) 

Jr2"'1 

0 

—  f(r  -t),/W/  x,/4-//  g  \x)d  x 
dt  6 

i 


dj(r-t) 

o 


(3.4.9) 
xh~H  dY  (x). 


Equation  (3.4.8)  then  follows  by  letting  g  -m.  To  prove  (3.4.9),  it  is  sufficient  to  show  that 
(cf.  (2.1.1)) 


g(t)  =  Cov{Y(t),  <j>(g)} 


=  E 


=  E 


[Y(t)-E{T(r)}K<t>(8)'E{0(g))] 


5//ir(0[<t>(8)-E{<|)(g))] 


However,  it  is  clear  that 


<t>(g)-E{<j»(g)}  = 


1 


r(3W/) 


2  r 


,2W-1 


'A-HZ'A-H  g'(x)dx 


dt\{t-x)*-Hx'A-HdBH,T{  X) 
0 


=  8V(Bwir)  (by  (3.3.16)). 


Hence, 
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•  jfiHI 


J 


E-{S//i7(r)[<K^)-E{<j)(^)}]  Cov-|Bw  ir(f),  (Bw tT,  g)n^ Bwir) 

•gif)  (by  (2.1.1)), 


and  (3.4.9)  follows.  ■ 

For  the  sake  of  completeness,  we  give  below  a  corollary  to  Theorem  3.4.1,  which  is  sim¬ 
ply  a  restatement  in  terms  of  the  signal  s  -m\  The  results  of  Theorem  3.4.2  are  already  in 
terms  of  m '  so  no  restatement  is  given. 

Corollary  3.4 J:  Problem  (3.4.1)  is  nonsingular  if  and  only  if,  for  almost  all  t  €  [0,7], 


s(0  =  Y(]hvi)  ~V4|(r-x)tf  s  W t. 


(3.4.10) 


where  3  e  L2([0,7]),  and,  for  almost  all  /  e  [0,7], 


r(3/z -H) 


(3.4.11) 


Proof:  Problem  (3.4.1)  is  nonsingular  if  and  only  if  (3.4.2)  is  nonsingular,  and  (3.4.2)  is 
nonsingular  if  and  only  if  m  satisfies  (3.4.4).  It  follows  from  the  proof  of  Theorem  3.3.4,  that 
m  satisfies  (3.4.4)  if  and  only  if  s  =  m'  satisfies  (3.4.10).  Equation  (3.4.11)  is  simply  a 
restatement  of  (3.4.5)  in  terms  of  s .  ■ 

Remark:  As  a  practical  consideration,  the  representation  of  the  likelihood  ratio  given  by 
Equations  (3.4.6)  through  (3.4.8)  is  probably  rather  difficult  to  implement.  One  would  prefer 
to  use  a  matched  filter  implementation,  if  possible.  This  is  easily  done  if  s  is  sufficiently 
well-behaved.  For  example,  suppose 
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T 

s(t )  =  VHH(2H-l)jh(x)\t-x\2H~2dx, 
o 

where  h  is  a  continuous  function  on  [0,7].  It  then  follows  immediately  that  the  likelihood 
ratio  is  given  by  (3.4.6)  with 

T 

(Y,  m)a(Xt  =  \h(x)dY{x\  (3.4.12) 

0 

and 

7T 

(m,  m)B( y)  =  VHH(2H-\)jjh(x)h(o)\x-a\w-2dxdo.  (3.4.13) 

oo 

Of  course,  in  this  case,  even  if  Y  is  non-Gaussian  (but  with  the  same  covariance  function 
Kb hit),  the  statistic  (Y,  m)a(Y)  given  by  (3.4.12)  is  the  linear  statistic  with  the  highest  signal* 
to-noise  ratio. 

3.5.  Detection  of  Gaussian  Signals  in  FGN 

Let  X  4  (X(f);  t  e  [0,7]}  be  a  process  observed  on  a  compact  interval  [0,7].  In  this 
section,  we  consider  the  following  hypothesis  testing  problem: 

Hq:  X  is  Gaussian  with  mean  zero  and  covariance  function  K  Bh  it 

versus  (3.5.1) 

Hr‘  X  is  Gaussian  with  mean  function  mj  and  continuous  covariance 
function  K j. 

This  problem  can  be  regarded  as  a  generalization  of  the  problem  of  detecting  Gaussian  signals 
in  white  Gaussian  noise,  which  has  been  widely  studied.  We  will  see  that  the  nonsingularity 
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conditions  for  the  detection  Problem  (3.5.1)  are  closely  related  to  the  following  well-known 
result  of  Shepp  [58]. 

Theorem  3.5.1:  Let  Xi  (X(f );  t  e  [0,7]}  be  a  Gaussian  process  and  let 
Kq (tyS)  =  min(r^)  for  e  [0,7].  (Recall  that  Kq  is  the  covariance  function  of  a  standard 
Brownian  motion  on  [0,7].)  Consider  the  following  hypothesis  testing  problem: 

HQ:  X  has  mean  zero  and  covariance  function  AT0 
versus  (3.5.2) 

Hjt  X  has  mean  function  mt  and  continuous  covariance  function  K j. 

This  problem  is  nonsingular  if  and  only  if  the  following  three  conditions  are  satisfied. 

(i)  There  exists  a  function  <)>  e  L2([0,7]  x  [0,7])  such  that 

IS 

{K  j  -  AT0)(r  ^ )  =  JJ<j>(T,o)d  odx,  [0,7  ] . 

00 

(ii)  If  o(d>)  represents  the  spectrum  of  the  operator  determined  by  the  function  0,  then 
-1  4  0(0). 

(iii)  There  exists  a  function  mj  6  L2([0,7])  such  that 

t 

mx(t)  -  V  t  e  [0,7]. 

o 

The  function  <j>  is  unique  and  symmetric  and  is  given  by 

<KM)  = -^[(tf1  - /W,*)1 

for  almost  all  (tj)  €  [0,7]  x  [0,7],  The  function  m  j  is  unique  and  :s  given  by 
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mj(r)  =  -^m(r) 

for  almost  all  t  e  [0,7]. 

Shepp  also  gives  formulas  for  the  likelihood  ratio  in  the  event  that  Problem  (3.5.2)  is 
nonsingular.  We  will  show  that,  if  Problem  (3.5.1)  is  nonsingular,  it  can  be  transformed  into 
an  equivalent  problem  of  the  form  (3.5.2),  in  which  case,  the  likelihood  ratio  can  be  found 
using  one  of  Shepp’s  formulas.  We  begin  by  giving  a  characterization  of  the  RKHS 

Theorem  3.5.2:  Let  the  set  of  functions  {/,;  r  e  [0,7]}  be  defined  by  ^  3.10).  Then 
H(KSH\T)®H(KBinT)  consists  of  functions  of  the  form 

n 

g  (f ^ )  =  J jft  (W s  (<r)§  <*d t,  t,s  e  [0,7],  (3.5.3) 

oo 

where  g  €  L2([0,7]  x  [0,7]).  Any  such  function  g  is  twice  differentiable  almost  everywhere 
in  [0,7]  x  [0,7]  and,  for  almost  all  (M)  6  [0,7]  x  [0,7], 

(3.5.4) 

Proof:  Recall  from  the  proof  of  Theorem  3.3.4  that  the  functions  (/r;  t  e  [0,7]}  span 
L2([0,7])  and 

T 

K B« ,r^ *  )  =  \ft  (U  Vs  (“  )du , 

0 


V  M  s  [0,7]. 
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It  follows  that  the  set  of  functions  (/,j)e[0,7]  x  [0,7]}  given  by 

/ (m)(VI)  4  ft  (x)/,  (c),  x,a  €  [0,7], 
spans  L2([0,7]  x  [0,7]),  and 

TT 

*B„ir('i.* i)* B„ir(^2^2)  =  JJ/ (,lJ2)(x,a)d cdx,  v  ti  e  [0,T]. 

00 


Since 


^((fi,f2)^i^2))  =  *BHir(fi^i)*B„ir(f2*f2)>  6  [0,7], 

is  the  reproducing  kernel  for  B(KB/iit)®B(KBhit),  it  follows  from  Theorem  2.1.1  that  a 
function  g  e  B(KBhit)  ®  H  (KBhit)  if  and  only  if  there  exists  a  function 
g  6  L2([0,7]  x  [0,7])  such  that,  for  all  tj  e  [0,7], 

TT 

g(tj)  =  jjf  (/iJ)(x,<j  )g  (x,a)d  ad  t 
oo 

TT 

-  jjft  Mfs  (<*)$  (x,a)d  od  x. 

00 


This  proves  (3.5.3).  To  establish  (3.5.4),  notice  that,  if  g  satisfies  (3.5.3),  then 
TT 

g(U)  =  jjf  t  (x)/,  (a)g  (x,c)d  ad  x 
oo 


1 

r(//-Vi) 

1 

T(H-x/2) 


2  Is 

til 


C*-WJuW~V-T)"-*d« 


al/t~H  Jvw_V4(v  -af-^dv 
a 


2 

jju“-\"-» 


00 


gTx~a)dad% 


UV 


J/x1^  (u  -<r)H-hgT^aJdadx 

00 


dvdu. 


It  follows  that,  for  almost  all  (tj)  e  [0,7]  x  [0,7], 
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=  f-prTrixr  ]  o'*-"  (f  -T)"-^  -<T)«  -^rfx^v  C-d  T. 

os  ot  1  (n  *  /i)  QQ 

Equation  (3.5.4)  follows  straightforwardly  by  using  the  above  formula.  ■ 

We  can  now  prove  the  following  theorem,  which  is  analogous  to  Shepp’s  result  for  sig¬ 
nal  detection  in  white  Gaussian  noise. 


Theorem  3.5.3:  Problem  (3.5.1)  is  nonsingular  if  and  only  if  the  following  three  condi¬ 
tions  are  satisfied. 

(i)  There  exists  a  function  $  6  L2([0,7]  x  [0,7])  such  that 


(K i  -  KSHtT)(tj)  =  jjft(x)fs(amx,a)d<ydx,  *  e  [0,7],  (3.5.5) 

oo 

The  function  0  is  real  and  symmetric,  and,  for  almost  all  (tj)  e  [0,7]  x  [0,7], 


<j>(f4)  = 


1  .H-Vi-H-V 4 

r?b-H) 

U  2  (3-5.6) 

_L_  {t^-H  (5  _<,)*-«  JL-(* ,  -  KBffiT)(X,0)dGdX. 


(ii)  If  a(<D)  represents  the  spectrum  of  the  operator  <5  determined  by  the  function  <J>, 
then  a(0)  c  (-1,°°). 

(iii)  There  exists  a  function  m(  6  L2([0,7])  such  that 


i  » 


m tit)  =  — lx*-"  juH-*(u^f^du  m}{x)dx ,  v  r  s  [0,7], 

l  (H  ~  n.)  q  T 


(3.5.7) 


The  function  m(  is  real,  and,  for  almost  all  t  e  [0,7], 


(3.5.8) 


*i(t)  =  t“-*-£  r(3^ fo-V™xA-ffm'1(x)dx  . 

Proof:  It  follows  immediately  from  Theorem  2.1.4  that  Problem  (3.5.1)  is  nonsingular  if 
and  only  if 

<0  (*i e  U <K»., ,>«»<*»,„>• 

(iiO  There  exist  constants  0  <  c  <  C  <  °°  such  that  cKB/f{T  c  K  ,  <  CAfBw)J.,  and 
(in')  m1  e  »(A:Bhit). 

Qearly,  by  Theorem  3.5.2,  condition  (i)  is  equivalent  to  (0,  and  it  follows  from  Theorem 
3.3.4  that  condition  (iii)  is  equivalent  to  (in')-  The  theorem  will  be  proven  if  we  can  show 
that  condition  (ii)  is  equivalent  to  (iiO-  To  this  end,  let  0  be  given  by  (3.5.6)  and  let  the 
linear  operator  <J>:L2([0,7])  — »  L2([0,7])  be  defined  by 

r 

(<Dg  )(r )  =  j0(r  ,x)g  (x )d x,  t  e  [0,7], 

o 

where  g  e  L2([0,7]).  The  operator  <I>  is  then  compact  and  self-adjoint  with  ||d>||  <  ||0||2. 
Further,  a(<D)  £  HI<1>||,  11^(1 1.  and  if  X  €  tf(d>),  then  X  =  0  or  X  is  an  eigenvalue  of  d>. 
Finally,  if  (XnJ^i  is  the  set  of  eigenvalues  of  O  and  the  associated  set  of  orthonor¬ 

mal  eigenfunctions,  then  zero  is  the  only  accumulation  point  of  {Xn}^l1;  [en }  “=1  spans 
L2([0,7]);  and 

0(f^)  =  'LXnen(t)e^rj,  VU£  [0,7], 

n=l 

where  the  above  convergence  is  in  L2([0,7]  x  [0,7]).  (These  standard  properties  of  L2 
integral  operators  are  discussed,  for  example,  in  [41].)  We  are  now  ready  to  show  that  (ii)  and 
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(ii7)  are  equivalent 

(ii7)  implies  (ii):  Suppose  there  exists  c  <0  such  that  (A^  -  cKBhit)  is  nonnegative 
definite,  and  let  X  be  an  eigenvalue  of  <t>  with  associated  (unit)  eigenfunction  e .  Since 
{/,;  t  e  [0,T])  spans  L2([0,T]),  there  exists  a  sequence  of  functions  of  the  form 

/n(x)  =  2ai/,.(t),  x  €  [0,T], 

i=l 

which  converges  to  e  in  L2([0,7']).  Now,  let  8  be  the  Dirac  delta  function  and  recall  that 

T  TT 

Kbhit(‘J)  =  lf,(x)fs(x)dx  =  fff,  (t)/,  (c)8(x-a)d  od  x. 

0  00 


Then, 


TT 

X  +  (1-c)  =  j\e(a)eTM4>(x,o)  +  (l-c)5(x-o)]dodx 
oo 


.* _ i 


[0(x,a)  +  (l-c)8(t-0)]dadT 


N 

=  Jim  X  ai  ai  (K  i -cK B„  IT)(ti  ,tj ) 

/  .y=l 


£0, 


and  it  follows  that  X  >  c-1  >  -1.  Hence,  c(<D)  £  (-1,  ||0(|  ]  and  (ii7)  implies  (ii). 

(ii)  implies  (ii7):  Now  suppose  that  o(O)  £  (-1,  |IO||  ].  Let  the  sets  {a,  }[L[  and  {r, 
be  arbitrary  and  define 

X«iA(t).  xe  [0,7]. 

i=i 


Then 
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N 


tt  r 


2>i//,(<*) 

1=1 


Lr=i 


1.7=1  00 

TT 

=  JJ*tf(o)*tf(x)[Wt,c)  +  (1  -c)8(x-a)]dodx 


[<j>(x,a)  +  (l-c)6(x~G))dadx 


00 


=  £[X,  +<l-c)!tvSw^»> 

n=l 


\|2 


>[inf  {^J-^  +  d-c)]  l^lb2 


>0  if  c  <  1  +  inf  {X* }  “=1. 


Since  -1  cannot  be  an  accumulation  point  of  the  set  we  always  have 

inf  {X„  }^t_!  >  -1,  and  we  can  find  0  <  c  <  1  +  inf  {X„  JjJlj  such  that  ( K !  -  cKBlnT)  is  nonne¬ 
gative  definite.  Similarly, 

N  TT 

£  aiaj(CKBHiT  -  KOHidj)  =  JJg*(o)£^[(C -l)S(T-a)  -  tfx,o)]dodx 

i  J=1  00 

=  £t(C-l)-X„]l^.e.)l2 

n= 1 

>  [(C-l)  -  i|<D||  ]  Hg^Hl 
>0  if  C  >  1  +  HOB. 


Hence,  we  can  find  0  <  c  <  C  <  <»  such  that  cKBf/IT  ATj  <c  CKBhit.  It  follows  that  (ii) 
implies  (ii'),  and  the  theorem  is  proved.  ■ 

We  now  wish  to  investigate  the  possibility  of  transforming  Problem  (3.5.1)  into  the  more 
familiar  form  (3.5.2)  considered  by  Shepp. 

Theorem  3.5.4:  If  Problem  (3.5.1)  is  nonsingular,  the  transformation 
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1  *  * 

2{t)  4  ---  Jr^d-fCT-ii )“"" u^dXiu ),  t  s  [0,7],  (3.5.9) 

r(  )  0  0 

is  well-defined  under  both  hypotheses.  The  integral  in  (3.5.9)  is  to  be  interpreted  as  an 
integral  with  respect  to  the  process  Y  4  {T(r);  t  e  [0,7]}  given  by 

t 

T(f)  4  — rj— •  f(r-t)^t^dX(x),  r  e  [0,7].  (3.5.10) 

r(3/2-//)  JQ 

The  new  observation  process  Zi  (Z(t);r  e  [0,7]}  is  Gaussian,  and  the  transformed 
hypothesis  testing  problem  becomes 

Hq:  Z  has  mean  zero  and  covariance  function  K0(tj)  =  min(rry). 
versus  (3.5.11) 

H{:  Z  has  mean  function  mz  and  covariance  function  Kz, 

where 

; 

mz(t)  =  jm^dx,  t  e  [0,7],  (3.5.12) 

o 

ts 

^z^ry)  =  |J<l>(i:,o)dadt  +  min(r^),  e  [0,7],  (3.5.13) 

oo 

and  <j>  and  m  \  are  given  by  (3.5.6)  and  (3.5.8),  respectively. 

Proof:  It  follows  from  Theorem  (3.3.5)  that,  under  Hq,  the  transformations  defined  by 

(3.5.9)  and  (3.5.10)  are  well-defined,  resulting  in  Z  being  a  standard  Brownian  motion  on 
[0,7].  Further,  since  Problem  (3.5.1)  is  assumed  to  be  nonsinguiar,  it  follows  that  (3.5.9)  and 

(3.5.10)  are  also  well-defined  under  Hj  (see,  for  example,  [7],  lemma  2.4).  So  we  need  only 
to  establish  that,  under  Hj,  Z  has  a  mean  and  covariance  given  by  Equations  (3.5.12)  and 
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(3.5.13),  respectively.  To  this  end,  let  Y  be  as  defined  as  in  (3.5.10)  and  notice  that,  under 
Hj,  Y  has  mean  function 


m Y(t)  =  — f(r-x)V4-// x*~Hm \x)dx  =  ix^Hmx(x)dx,  t  e  [0,7], 
!(•%—//  )  JQ  o 


(3.5.14) 


and  covariance  function 


Ky(‘J)  = 


Clearly, 


1 


r(3/2-//) 


2  is 


ff(r x^H  (s  -a)l/*-H  d^H  -^~Kx(x,a)d  ad x,  rj  e  [0,7 1 

oo  d<ydx 


Kx(tj)  =  (K j 


and  it  follows  from  the  proof  of  Theorem  3.3.5  that 

12  is 


^min^^^l  1 


n3/z-H) 


a2 


Jf(r  -x)l/W/ x*-«  (5  -a)1/i_//  -j-t-K  Bw  ir  (x,a)d  ad  t 

00  dOdt 


Thus,  using  these  relationships  and  (3.5.6),  we  get  straightforwardly  that 


ts 


=  \[x,A~H d/*~H $(x,a)dadx  +  -  1  ■  min(r2  w  j2  2H),  s,t  e  [0,7]. 
oo  2—2H 

(3.5.15) 

Hence,  we  have  established  that,  under  H1?  process  Y  has  a  mean  and  covariance  given  by 
(3.5.14)  and  (3.5.15),  respectively.  Now,  writing  Z  as 


Z(/)  =  Jx//-%dY(x),  t  e  [0,7], 


and  proceeding  in  a  manner  similar  to  the  above,  it  follows  straightforwardly  that,  under  Hr 
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Z  has  a  mean  and  covariance  given  by  (3.5.12)  and  (3.5.13),  respectively.  This  establishes 
the  theorem.  ■ 

We  have  shown  that,  if  Problem  (3.5.1)  is  nonsingular,  (3.5.9)  can  be  used  to  transform 
Problem  (3.5.1)  causally  into  the  form  (3.5.2),  which  corresponds  to  the  problem  of  detecting 
a  Gaussian  signal  in  white  Gaussian  noise.  Furthermore,  since  (3.5.9)  is  invertible  under  H0 
(via  (3.3.11)),  the  two  problems  are  equivalent.  Since  Shepp  has  given  formulas  for  the  likeli¬ 
hood  ratio  for  Problem  (3.5.2),  we  have  a  characterization  for  the  optimal  solution  to  Problem 
(3.5.1). 

As  a  final  consideration,  we  restrict  our  attention  to  a  special  case  of  Problem  (3.5.1).  In 
particular,  we  assume  that,  under  H2,  X  is  Gaussian  with  m1  -  0,  and 

i  **  -iojf  i  pi (os  i 

~  J  S^(o)d(0,  vue  [OJ],  (3.5.16) 

2k  J"  1(0  l  (0 

where  S  x  is  a  spectral  "density"  satisfying 

lim  inf  l©lnSi(a))  >  0,  (3  5  171 

I  CO  I  — 

and 


Sj((0) 

JL  (1  +  C02)n 


I 


■d  0)  < 


(3.5.18) 


for  some  integer  n .  The  restriction  that  =  0  is  not  really  necessary,  merely  convenient, 
while  conditions  (3.5.16)  through  (3.5.18)  simply  imply  that,  under  Hj,  the  observed  process 
has  stationary  increments  with  well-behaved  spectral  densities.  (See  [77]  and  [78]  for  a  dis¬ 
cussion  of  processes  with  stationary  increments.)  These  conditions  are  not  very  restrictive, 
and  when  they  hold,  the  observation  process  X  can  be  defined  as 
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X(t)  =  jW{x)dx  A  W(/[0>t]),  t  €  [0 ,T], 

o 


where  W  is  a  generalized  stationary  Gaussian  process. 

If  we  let  SwH  be  the  spectral  density  of  FGN,  that  is, 

SWh(oj)  =  lo)l1_2//,  O*coe0?, 


then  we  can  rewrite  Problem  (3.5.1)  in  the  following  equivalent  form: 


Hq:  W  has  spectral  density  5  ww 


versus 


Hjt  W  has  spectral  density  Si- 


With  regard  to  this  formulation,  we  have  the  following  result 
Proposition  3.5.5:  If  there  exits  (3  >  2 H  -  lA  such  that 


Urn  lo)|P  [S^co)  -Sw„(g>)]  =  0, 

I  00 1  — 


then  (3.5.19)  is  nonsingular.  If 


lim  inf  [Si(cu)  -  5w„(o>)l  >  0, 

I  y.vt  . _  H 


then  (3.5.19)  is  singular. 


(3.5.19) 


The  proof  of  Proposition  3.5.5  is  essentially  due  to  Rozanov.  For  W  a  stationary  L2- 
process  with  integrable  spectral  densities,  the  result  is  proven  in  [56].  The  extension  to  the 
case  considered  here  follows  easily  from  Lemmas  8  and  12  in  [57]. 
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For  a  wide  class  of  covariance  functions  K\,  this  result  gives  simple  nonsingularity  con¬ 
ditions  for  Problem  (3.5.1).  For  example,  if,  under  Hj, 

X{t)  —  Bu>\j{t)  +  Bn\i{t),  t  €  [0,7], 

where  i  [BH^T(t)-,  t  s  [0,7]}  is  another  FBM  independent  of  Bw  ,7-,  then  Problem 

(3.5.1)  is  nonsingular  if  and  only  if  H'  >  H  +  V*.  Heuristically,  Proposition  3.5.5  implies  that 
a  Gaussian  signal  produces  a  nonsingular  shift  in  additive  FGN  when  the  signal’s  spectrum 
has  o(l/  tails. 


3.6.  Performance  Characteristics 

In  this  section,  we  investigate  some  aspects  of  detector  performance  in  the  presence  of 
additive  fractional  Gaussian  noise.  In  particular,  we  derive  the  signal-to-noise  ratios  of  the 
optimal  detectors  for  a  class  of  baseband  signals.  We  also  derive  channel-mismatch  degrada¬ 
tion  factors  for  two  members  of  this  class. 

Consider  the  following  normalized  version  of  Problem  (3.4.1): 


where  the  normalizing  factor  c/yJVH  reflects  the  assumption  that  the  noise  has  "power"  a2 
(i.e.,  Var{K(l)}  =  a2).  It  follows  from  Corollary  3.4.3  that  Problem  (3.6.1)  will  be  nonsingu¬ 
lar  if  and  only  if,  for  almost  all  t  e  [0,7], 


54 


s(t)  = 


>/VtfT XH-U)  o 


(3.6.2) 


where  J  e  L2([0,J]),  and,  for  almost  all  t  e  [0,T], 


3(f) 


_  ^  d 

~  are/2r-H)  dt 


J(r  -  x1/W/  s  (x)d  x 


(3.6.3) 


Assuming  nonsingularity,  the  log-likelihood  ratio  for  (3.6.1)  is  then  given  by  (of.  (3.4.6)) 


where 


log  L(Y)  =  (Y,  m) jky)  -  V4  (m,  m) jf  (Y)» 


(3.6.4) 


and 


m(f)  i  Js(x)dx, 


(nt,  m)a(Y)  =  J[3  (x)]2dx, 
o 


(v.  "Wro  ■ 


(3.6.5) 

(3.6.6) 


(3.6.7) 


It  follows  from  (2.1.2)  and  (2.1.3)  that  log  L(Y)  is  distributed  in  the  following  manner 
under  the  two  hypotheses: 


Hq.  log  L(Y)  -  N  j-Vi  (m ,  m)m Y),  [m ,  m)W(Y)  j, 
Hj.  log  L(Y)  -  N  £  Vi.  (m ,  (m ,  m)a (Y)j. 


Assuming  equal  prior  probabilities  on  Hq  and  Hj,  the  minimum- probability-of-error  detection 
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rule  takes  the  form  (see  [54]) 


1  if  log  L(Y)  >  0 

5(Y)  =  • 

IP  if  log  L(Y)  <  0 


and  the  minimum  probability  of  error  is  Ptf*  =  Y)  )*  where  <t>  is  the  standard 

normal  distribution  function.  Note  that  P*  is  a  decreasing  function  of  y),  which 

represents  the  signal-to-noise  ratio  (SNR)  of  the  optimal  detector  for  Problem  (3.6.1). 

Now,  suppose  that  the  signal  s  is  of  the  form 


'J'QL 


t  €  to ,n 


(3.6.8) 


where  C  is  any  real  constant  and  a  k  0.  All  of  these  signals  have  average  power  C2  and 
satisfy  (3.6.2)  with  s  given  by 

»  ■ 

oTar(3/2-ff)  dt  J0 


oTan3/2-//)  dt  A 


_  CyjVH  (2a+l)  d  fa+2-2 H  xa+'/t-H dx 

cfTar(3/2-H)  dt  { 

_  C(ct+2-2H)y]VH(2a+\)  t,x_^Ar-Hx a+^H dx 

aTT(3/2-//)  6 


C  (a+2-2H  hjVH  (2a+l)  ^  T(3ti-H)r(a+3l2-H) 

cTaT(3/2~H)  1  T(a+3-2 H) 


C  T(a.+3t2-H  )^VH  (2a-+l ) 

aTar(a+2-2//) 


ta+'A-H 
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Hence,  the  SNR  of  the  optimal  detector  is 


(m,  m)B(Y)  =  J[2  (x)]2dx 


C2Vh(  2a+l) 

o2(2a+2-2//) 


T(a+2/2-H) 

T(a+2-2H) 


? 

7-2-2 H 


(3.6.9) 


For  H  -  Vi  expression  (3.6.9)  reduces  to 

'm  m x  -  C2  r 

as  one  would  expect,  since  H  =  x/i  corresponds  to  white  noise.  In  this  case,  the  optimal  SNR 
grows  linearly  with  respect  to  the  length  of  the  observation  interval.  However,  for  any  H  in 
the  range  xh<H  <  1,  (m,  grows  sublinearly  with  respect  to  T .  In  fact,  for  H  ~  1, 

the  performance  of  the  optimal  detector  is  virtually  independent  of  the  length  of  the  observa¬ 
tion  interval.  This  is  in  spite  of  the  fact  that  the  signal  energy  grows  linearly  with  T  and  is  in 
marked  contrast  to  the  asymptotically  linear  growth  that  one  would  expect  if  the  noise  process 
had  a  rational  spectrum. 

It  is  interesting  to  note  that  a  constant  signal  (a  »  0)  is  essentially  the  only  periodic  sig¬ 
nal  that  displays  this  behavior.  More  precisely,  if  s  is  a  continuously  differentiable  periodic 
signal,  the  optimal  SNR  will  grow  asymptotically  at  a  linear  rate  for  all  H  in  the  range 
V2  <l  H  <1.  The  proof  of  this  fact  is  straightforward,  although  somewhat  messy,  and  is  omit¬ 
ted. 

For  fixed  T,  the  behavior  of  the  optimal  SNR,  as  a  function  of  a  and  H ,  is  pretty  much 
as  one  would  expect,  that  is,  increasing  in  both  a  and  H.  In  fact,  (3.6.9)  implies  that,  for 


fixed  T , 
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and 


lim  (m,  mVfvi  =  «>,  V  a  >  0, 
h-*i  N 


lim  (m,  ^i)n(Y)  =  °°,  V  H  >  xh. 


The  behavior  when  a  =  0  is  interesting.  In  this  case,  expression  (3.6.9)  reduces  to 


/_  _A  C2T(3/2-H)  t2-2H 

X  ’  //r(Y)~  <PlHT{H+xA)TQ-2H) 


and,  as  H  1,  we  get 

/  \  c2 

\m’  m/B  (Y) 

An  optimal  SNR  equal  to  C 2l<s2  corresponds  to  the  case  in  which  the  sample  paths  of  the 
noise  process  are  constant  with  probability  one.  In  fact,  if  we  were  to  extend  the  definition  of 
FGN  (with  power  a2)  to  include  the  limiting  case,  H  =  1,  we  would  want  just  such  a  process 
since 


lim  -tt-Kw  (x)  =  lim  g2H(2H-\)\x\2H~2  =  a2, 
H  — *1  VH  ""//_»! 


V  X  6  R. 


Hence,  heuristically,  FGN  can  be  regarded  as  going  from  a  completely  unpredictable  process 
at  H  =  xh  to  a  completely  predictable  process  at  H  =  1. 

Now  suppose  that  s  is  of  the  form  (3.6.8)  but  that  H  is  incorrectly  identified;  that  is, 
suppose  that  log  L(Y)  is  computed  using  (3.6.3)  -  (3.6.7)  on  the  basis  of  H'  *H.  Then 
log  L(Y)  (which,  in  this  case,  is  not  the  true  log-likelihood  ratio)  is  distributed  as  follows: 
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Hq.  log  L(Y)  -  N(-V4*i,  v2), 

H1:  log  L(Y)  -  N(  v2), 

where  (using  (3.6.9)) 

n  -  e,-  ■ 

> 

cr(  i2—h  )  q  o 

(3.6.10) 

r 

=  J[3(x)]2dt 

o 

_  r(q4-3/2— //  0  J-2-2H' 

~  o2(2a+2-2// )  [r(a+2-2//0 


and 
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v*  =  Var 

or(  h.  h  ) «  0 


Var  — -  - - - frad, f(r-x)^w  dS„  ,r(x) 

aVv^Tar  (OL+2-2H  ^Wi-H  Vo  o 


CV^TCa+^-Z/OV^a+T  *  tA_„, 

Var- — ■  -■ - - -  Ta  (f-x)A-w  x^  dBwir(x) 

aVvvrar(a+2-2H')r(3/2-z/')  i 


-  a  J/a“l  J(r-t)14_w'x14_///dfiw  |T(t)  dr  ■. 
o  o 


(3.6.11) 


We  consider  two  special  cases. 

Case  1:  Constant  Signal  (a  =  0) 

If  a  =  0,  expression  (3.6.10)  can  be  rewritten  as 

=  C2VHWh-H\'h-H-)  T,_m. 

^  ~  <slT(2-2H’} 

where  P  represents  the  beta  function,  and  (3.6.1 1)  reduces  to 

(T  I  x-x>  1 2W_2d  x  d  v> 

'*~H  ’jhr-H '( 1  _^h-H  '^h-H '  j  x_u  1 2//  -2d  xd  v 


cvH . 


- H  {2H  -1)  [f(T 

aT(2-2H')  J}J; 


CVH’ 

or(2-2W0 


H  (2 H  -i)72+2W  'JJ(  i  -t) 

00 


5r^§jn  H ji 


where 
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e(Htf  0  4  JJ(l-t)'^w 0-u)'^ V^' I t— u I 2H~1dxdM. 
oo 


Hence,  the  probability  of  error  if  a  =  0  and  H'  *  H  is 


P*  =  d>  -  JL 
*  2v 


=  o  ri-tf 


2<rlH(2H-l)e(Htf') 


Naturally,  letting  H'  =  H  gives  us  the  minimum  probability  of  error,  that  is. 


P*  =  <D(-‘/W [m,  m) B(Y) ) 

-  ^  f  -C3(3/2-//,3/2-//) 

2<rlH  (2H -\)e  (H  JH) 


It  follows  that 


*  2<rW(2ff-l)e  (//,//') 

•  _ 


<?(////')  P(3/2-//,  3/2-//) 


2<rJH(2H-l)e(H,H) 


=  &  _./2A//  X  g(H^)  P(3/2-//f,  3/2-//') 

V  X  ’  <Y>  e  (//  //  0  (3(3/2-//  ,  3/2 — //  ) 


Hence,  the  degradation  factor  8(H  Ji '),  which  represents  the  loss  in  SNR  due  to  channel 


mismatch,  is  given  by 


[p(3/2-///,  WO? 

*  ^  g(////0  [  (3(3/2-// ,  3/2-// )  • 


The  above  derivation  was  carried  out  assuming  that  both  H  and  H'  were  contained  in 


the  interval  (V4,l).  However,  it  is  straightforward  to  show  that  all  of  the  above  holds  if 
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H'  =  V 2  and  Vi  <  H  <  1;  that  is,  if  a  nonwhite  channel  is  assumed  to  be  white.  If  the  actual 
channel  is  white,  however,  the  expression  for  v2  takes  a  different  form.  It  is  easy  to  see  that, 
if  H  =  Vi  and  Vz  £  //'  <  1,  we  get 

2  1 
0 

2 

T^h '3(2-2//',  2-2/0- 


In  this  case,  the  degradation  factor  is  given  by 


s<v*,/n  = 


[  S(3/2-/Q  3/2-/Q  I2 
j3(2-2// ',  2-2//  0 


We  have  evaluated  these  degradation  factors  numerically  for  various  values  of  H  and 
//'.  The  results  are  presented  in  Table  3.6.1.  As  this  table  shows,  if  the  channel  is  actually 
white  (H  ~  XA),  the  detector  performance  is  very  sensitive  to  mismatch.  In  fact,  as  H '  — >  1 , 
we  have  h{VzJ1')  — >  0.  On  the  other  hand,  Table  3.6.1  shows  that,  if  the  actual  channel  is 
nonwhite  (//  >  V4),  the  detector  performance  is  fairly  insensitive  to  mismatch.  In  fact,  if 
H  ~  1,  there  is  virtually  no  loss  in  performance  regardless  of  what  value  of  H'  is  used  to 
design  the  detector. 

Clearly  then,  in  the  constant  signal  case,  there  is  little  to  be  gained,  and  potentially  much 
to  be  lost,  by  using  a  detector  designed  to  match  an  assumed  value  of  H  >  Vz.  The  robust 
strategy  is  to  use  a  simple  integrate-and-dump  detector.  This  behavior  is  apparently  related  to 
the  fact  that,  when  using  constant  signals  on  a  channel  for  which  H  >  Vz ,  both  the  signal  and 
noise  power  are  concentrated  at  the  origin  in  the  frequency  domain.  As  we  show  below, 


Table  3.6.1.  Degradation  8(//  fl')  Due  to  Mismatch  (Constant  Signal) 


H' 

H 

0.500 

0.600 

0.700 

0.800 

0.900 

0.950 

0.975 

0.999 

0.500 

1.0000 

0.9910 

0.9528 

0.6140 

0.3823 

0.2160 

0.0098 

0.600 

0.9950 

1.0000 

0.9962 

0.9622 

0.8716 

0.7850 

0.7247 

0.6533 

0.700 

0.9864 

0.9960 

1.0000 

0.9645 

0.9341 

0.9124 

0.8862 

0.800 

0.9815 

0.9902 

0.9970 

1.0000 

0.9950 

0.9809 

0.9734 

0.900 

0.9847 

0.9898 

0.9946 

0.9984 

1.0000 

0.9994 

0.9986 

0.9972 

0.950 

0.9906 

0.9932 

0.9960 

0.9982 

0.9998 

1.0000 

1.0000 

0.9996 

0.975 

0.9948 

0.9962 

0.9976 

0.9988 

0.9998 

1.0000 

1.0000 

1.0000 

0.999 

0.9998 

0.9998 

0.9998 

1.0000 

1.0000 

1.0000 

1.0000 

1.0000 

when  using  nonconstant  signals,  there  is  much  more  to  be  gained  by  properly  identifying  the 
self-similarity  parameter  H . 


Case  2:  Ramp  Signal  (a  =  1) 


For  a  =  1,  (3.6.10)  and  (3.6.11)  become 


4  = 


3c2y 

o2(4-2//  ') 


T(5/2-H') 

T0-2H') 


-2-2 H" 


’2 
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V2- 


V3cvw^r(5/2-//o 

arr(3-2W')r(3/2-//o 

r 


2 

H  (2H -l)T4+2H~4H' 


11 


•  •  i  X-D  1 »** 

00 


2(1-d) 
3/2 -//' 


+ 


(l-pq-D) 

(3/2-// 02 


dtdl)  f 


^3CVhT(5/2-H') 

ar(3-2/nr(3/Wh 


2 

//  (2//-l)T2+2//-4//'e  (//  ,// '), 


where 


li 


*(//,//')  £  JJd-x^V^'d-^^'-o^'lx-Dl2"-2 

00 


2(1 -u) 
3/2-//' 


+ 


(1-XXl-D) 

02-H')2 


dxdv. 


Proceeding  as  before,  we  find  that,  for  Vz<  H  <1  and  l/z  £  H’  <  1, 


g  ,  _  r&z-HTWi-H^rO-lHKA-lH)  I  e(H,H) 

K  *  }  ~  r(5/2-H  )n3f2-H  )r(3-2H  0(4-2//  0  V  e  (//,//  O' 


Further,  for  H  =  Vi  and  lA  £  H'  <  1,  it  is  straightforward  to  show  that 


8O/2,// 0  = 


r(5/2-//  on3/:-//  o  /  3 

r(3-2//')(4-2//')  >  *(fc,//')  ’ 


where 


e(lAJH')  ±  (3(2-2// ',  2-2// 0 


3^7  (3(3-2//',  2-2// 0  +  —  — (3(4-2// ',  2-2//'). 
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Again,  we  have  evaluated  these  degradation  factors  for  several  values  of  H  and  H'.  The 
results  are  presented  in  Table  3.6.2.  As  this  table  shows,  detector  performance  continues  to 
be  quite  sensitive  to  channel  mismatch  when  H  =  l/t.  However,  with  this  signal  structure, 
detector  performance  is  also  quite  sensitive  to  mismatch  when  H  >  V4.  This  would  seem  to 
indicate  that,  for  general  nonconstant  signals,  significant  performance  improvements  can  be 
expected  due  to  proper  identification  of  the  parameter  H . 


Table  3.6.2.  Degradation  8(//  tf')  Due  to  Mismatch  (Ramp  Signal) 


H' 

H 

0.500 

0.600 

0.700 

0.800 

0.900 

0.950 

0.975 

0.999 

0.500 

1.0000 

0.9712 

0.8505 

0.5964 

0.1129 

0.0514 

0.0019 

0.600 

0.9783 

1.0000 

0.9600 

0.7896 

0.4791 

0.3109 

0.2357 

0.1722 

0.700 

0.9084 

1.0000 

0.9330 

0.6745 

0.4908 

0.3996 

0.3179 

0.800 

0.9415 

1.0000 

0.8701 

0.6912 

0.5845 

0.4808 

0.900 

0.5081 

0.5796 

0.6901 

0.8588 

1. 0000 

0.9216 

0.8188 

0.6912 

0.950 

0.2980 

0.3453 

0.4272 

0.5864 

0.8906 

1.0000 

0.9561 

0.8308 

0.975 

0.1623 

0.1896 

0.2392 

0.3476 

0.6498 

0.9281 

1.0000 

0.9136 

0.999 

0.0071 

0.0083 

0.0107 

0.0165 

0.0419 

0.1214 

0.3341 

1.0000 
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3.7.  Conclusion 

In  this  chapter,  we  have  considered  the  problem  of  detecting  signals  in  the  presence  of 
additive  fractional  Gaussian  noise.  We  have  applied  results  from  the  theory  of  reproducing 
kernel  Hilbert  spaces  to  give  necessary  and  sufficient  conditions  for  the  problem  to  be  non¬ 
singular  and  to  develop  whitening  filters.  For  the  case  of  a  stationary  stochastic  signal,  we 
have  interpreted  these  results  in  the  frequency  domain.  In  the  case  when  the  signal  is  deter¬ 
ministic,  we  have  characterized  the  optimal  detector  in  terms  of  the  likelihood  ratio.  Finally, 
we  have  studied  some  aspects  of  detector  performance  on  FGN  channels. 

An  interesting  problem  that  we  have  not  addressed  in  this  context  is  that  of  sequence 
detection  in  FGN.  It  is  obvious  that,  in  the  presence  of  such  strongly  dependent  noise,  the 
use  of  any  one-shot  strategy  (optimal  or  not)  to  detect  a  sequence  of  signals  will  lead  to  a 
sequence  of  strongly  dependent  detection  errors.  The  study  of  this  phenomenon  and  its  conse¬ 
quences  is  an  interesting  topic  for  further  investigation. 
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CHAPTER  4 

AN  RKHS  APPROACH  TO  ROBUST  DETECTION  AND  ESTIMATION 


4.1.  Introduction 

In  this  chapter,  we  consider  the  problems  of  L1  estimation  and  signal  detection  in  the 
presence  of  uncertainty  regarding  the  statistical  structure  of  the  random  processes  involved. 
The  classical  approach  to  designing  estimation  and  detection  procedures  relies  upon  exact 
knowledge  of  this  statistical  structure,  and  procedures  designed  to  be  optimal  for  a  particular 
nominal  structure  sometimes  perform  very  poorly  if  the  actual  statistical  structure  varies  even 
slightly  from  the  nominal.  The  minimax  approach  to  designing  robust  procedures  that  display 
some  amount  of  tolerance  to  variations  in  the  actual  statistical  structure  of  the  problem  has 
been  studied  in  recent  years  by  many  authors.  The  work  presented  here  is  most  closely 
related  to  and  motivated  by  the  results  presented  in  [52],  [53],  [66],  and  [70].  Other  related 
results  can  be  found,  for  example,  in  [9],  [10],  [24],  [39],  [40],  [65],  [68],  and  [69]. 

We  formulate  and  analyze  the  robust  estimation  and  detection  problems  in  the  context  of 
reproducing  kernel  Hilbert  space  theory.  Although  the  relationship  between  classical 
detection/estimation  and  RKHS  theory  is  well-known  (see,  for  example,  [22],  [23],  and  [48]), 
this  theory  has  not  been  applied  previously  to  the  study  of  robust  estimation  and  detection. 
By  using  an  RKHS  approach,  we  are  able  to  generalize  the  notion  of  a  linear  filter  and  to  give 
necessary  and  sufficient  conditions  for  such  a  filter  to  be  robust  in  the  minimax  sense  for  the 
general  L  Estimation  problem  in  which  there  is  uncertainty  in  both  the  covariance  structure  of 
the  observed  process  X  4  {X(f)l  and  the  cross-covariance  structure  of  X  and  Z,  the  variable 
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to  be  estimated.  We  show  that,  under  mild  regularity  conditions,  the  robust  filter  can  be 
found  by  solving  a  related  minimization  problem.  We  also  give  conditions  sufficient  to  insure 
that  the  robust  filter  exists.  In  particular,  we  show  that,  if  the  covariance  structure  of  the 
observed  process  is  assumed  to  be  known,  so  that  the  only  uncertainty  is  in  the  cross- 
covariance  structure  of  X  and  Z,  then  a  robust  filter  will  always  exist  and  can  be  found  by 
solving  a  straightforward  minimization  problem. 

A  somewhat  surprising  consequence  of  this  analysis  is  the  striking  similarity  between 
these  results  for  the  general  robust  L2-estimation  problem  and  results  given  by  Poor  in  [53] 
relating  to  robust  matched  filtering.  Reformulating  the  robust  matched  filtering  problem  in  an 
RKHS  context  allows  us  to  extend  Poor’s  results  and  clearly  reveals  the  underlying  similarity 
between  the  robust  estimation  and  matched  filtering  problems.  In  fact,  the  structures  of 
minimax  solutions  to  the  two  problems  are  seen  to  be  virtually  identical. 

As  a  final  application  of  the  RKHS  approach  to  robustness,  we  consider  the  problem  of 
robust  quadratic  detection  of  a  Gaussian  signal  in  the  presence  of  Gaussian  noise,  in  which 
the  deflection  ratio  is  used  as  a  performance  criterion.  We  show  that  this  problem  also  can  be 
formulated  in  an  RKHS  context,  and,  when  the  structure  of  the  noise  covariance  is  assumed  to 
be  known,  is  exactly  analogous  to  the  robust  matched  filtering  problem.  If  the  covariance 
structure  of  the  noise  is  also  unknown,  the  robust  quadratic  detection  problem  can  be  embed¬ 
ded  in  a  larger  problem,  which  is  again  analogous  to  the  robust  matched  filtering  problem.  A 
robust  filter  for  this  larger  problem  will  then  possess  desirable  robustness  properties  when 
applied  to  the  quadratic  detection  problem. 

This  chapter  is  organized  as  follows.  In  Section  4.2,  we  present  some  definitions  and 
notation  that  will  be  used  throughout  the  chapter.  In  Section  4.3,  we  discuss  the  general 
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robust  L  Estimation  problem,  and  in  Section  4.4  the  robust  matched  filtering  problem. 
Robust  quadratic  detection  is  discussed  in  Section  4.5.  Section  4.6  presents  some  concluding 
remarks. 

4.2.  Preliminaries 

Throughout  this  chapter,  X  ^  {X(r);  t  e  /}  will  represent  an  observed  process  defined 
on  an  arbitrary  index  set  /.  All  random  variables  will  be  assumed  real-valued  with  mean  zero 
and  finite  variance  unless  otherwise  specified.  The  extension  of  all  results  to  the  case  of 
complex-valued  random  variables  is  straightforward. 

Suppose  that  AT  is  a  class  of  covariance  functions  defined  on  an  index  set  / .  We  define 
B (IK)  as  the  set  of  all  functions  belonging  to  B(K)  for  some  K  e  K;  that  is, 

B(K)  =  kj  B(K). 

K  e  K 

Note  that  B  (K)  is  generally  not  an  RKHS,  nor  is  it  necessarily  closed  under  addition.  How¬ 
ever,  if  K  is  a  convex  set  of  covariance  functions,  then  B(JK)  is  convex.  This  follows  from 
the  fact  that  if  r0  €  B(K0)  and  e  B(K{),  then  (l-a)s0  +  asj  €  H ((l-a)^0  +  otXi)  (see 
[1],  §1-6). 

We  define  a  finite  filter  on  K  as  a  pair  h  =  ({hi }  "=1,  (fjl/Li),  where  n  is  a  positive 
integer,  {A,}/! Lt  cl?,  and  {rf}/L j  £/.  For  each  K  e  K ,  the  function  UK  €  B(K)  is 
defined  by 

hAT( )  4  ZhiKi-Ji). 

i=i 


Similarly,  we  define  a  filter  on  K  as  a  sequence  h  =  {h^}jv=i  of  finite  filters  such  that,  for 
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every  K  g  K,  the  sequence  }jv=1  c  B(K)  converges  in  B(K).  For  each  K  g  IK,  the 

function  fix'  e  B(K)  is  then  defined  as 

A  _  nm 

Mfoi  lim  h„K(-)  =  Iim  £  h^K^^N)), 

A/— **•  V— j_ j 

where  the  limit  is  taken  in  IF  (AT)  (and  consequently  pointwise,  as  well).  In  order  to  simplify 

«• 

notation  somewhat,  we  will  regard  as  a  single  sequence  {r, j  and  write 

hAT()  =  lim 

i=i 

where  it  is  understood  that,  for  each  N ,  the  sequence  {/r,*(AO}£i  has  only  finitely  many 
nonzero  elements  corresponding  to  the  appropriate  values  of  the  sequence  {/,•  )“j. 

For  any  class  K  of  covariance  functions  on  /  ,  we  denote  by  IF  (IK)  the  class  of  all  filters 

on  K,  and  we  note  that  the  class  of  finite  filters  can  be  regarded  as  a  subset  of  F (IK).  To 

denote  the  set  of  filters  defined  on  the  singleton  class  K  =  {AT},  we  simply  write  F(K);  thus, 

F(K)  =  n  F(K).  For  any  h  e  F(K),  we  can  define  a  function  (•,  h) :  B (IK)  — >  IR  in  the 
KeK 

following  manner.  Let  K  €  K  and  s  g  B(K).  Then 

(s ,  h)  4  is ,  hK)a  (£) 

“  \s  >  ^nK)b(K) 

=  \s  >  ^NK)a(K) 

=  lim  [s,  £/ii(lV)A:(-,rI))Ji(jn 

N~*~  i=i 

=  lim  ZkiWMi). 

N~>°°  i= i 

It  is  clear  that  h)  is  well-defined  on  B  (IK)  for  any  he  F(K).  Further,  for  any  h  g  F  (IK) 
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and  AT  e  K,  the  restriction  of  (■,  h)  to  H (AT)  defines  a  bounded  linear  functional  on  U(K). 
Also,  for  any  K  €  K,  we  have 

(hAT,  h)  =  (hAT, 

=  lim  (h^AT,  h\Ar)a(X) 

=  lim  £  hj  (N  )h:  (N)K  (r .-  ,tj ). 

»,;=1 

If  the  process  X  has  covariance  function  ATX  e  IX,  then,  for  each  h  6  IF  (IK),  we  can 
define  h(X)  e  L2(X)  as 

h(X)  ^  lim  h*,(X) 

N->~ 

.  -  (4.2.1) 

4  lim  ('«)> 

«-»•  **i 

where  the  limit  is  taken  in  the  mean-square  sense.  The  fact  that  this  limit  is  well-defined  fol¬ 
lows  from  the  congruence  between  If  (Afx)  and  L2(X);  that  is, 

lim  E([h„(X)-h„(X)]2)=  lim  liS^X,  -  hMArxll|«,, 

N  M  >°*  /V  jw  >**• 

=  o. 

Note  that  (4.2.1)  implies  that  h(X)  =  (X,  hAT  x)n  (jcx)- 

Finally,  the  term  signal  will  usually  refer  to  a  function  s:  I  — >  IR ,  and  uncertainty  class 
will  always  mean  a  set  of  signal-covariance  pairs  (sJC).  The  term  admissible  uncertainty 
class  refers  to  an  uncertainty  class  U  with  the  following  additional  properties: 

(i)  for  all  pairs  (sJC),s  e  H(K),  and 

(ii)  U  is  convex;  that  is,  if  (sqJCq)  e  U  and  (s^X^e  U,  then 

((l-a)s0  +  05!,  (l-a)X0  +  aKi)  e  U  for  all  0  £  a  £  1. 
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Given  any  uncertainty  class  B ,  we  associate  with  it  the  class  K{B )  of  covariance  functions 
"contained  in"  U ;  that  is. 


K(B)k 


SAT:  (sJC)  e  B  for  some  signal  s  r. 


We  are  now  ready  to  discuss  the  problem  of  robust  L 2  estimation. 


4.3.  Robust  L 2  Estimation 

Let  X  4  {AT(f);  t  e  I }  be  an  observed  zero-mean  process  with  covariance  function  Kx, 
and  let  Z  be  an  arbitrary  zero-mean,  L 2  random  variable.  If  Z(X)  is  the  projection  of  Z  onto 
L2(X),  then  we  know  that 

E{[Z -Z(X)]2}  =  min  E {[Z-U]2}.  (4.3.1) 

UeLHX) 

That  is,  Z(X)  e  L2(X)  is  the  linear  estimate  of  Z  that  minimizes  the  mean  squared  error 
(MSE).  Of  course,  Z(X)  is  also  the  unique  element  of  L2(X)  that  satisfies  the  equation 

E{X(r)Z }  =  E{X(t)Z(X)),  V  r  e  /.  (4.3.2) 

We  can  restate  these  relationships  in  RKHS  terms  by  defining  Sx'  I  — >  IR  as 

sz(0  4E{X(r)Z},  *tel. 

It  then  follows  from  Equations  (2.1.1)  and  (4.3.2)  that  sz  €  B  (ATX)  and  (x*  sz)a(Kx)  ~ 
Hence,  if  ATX  and  sz  are  known,  the  linear  estimate  Z(X)  e  L2(X)  satisfying  (4.3.1)  can  be 
determined  as  follows.  Since  s2  e  H(KX),  there  exists  a  filter  he  IF (K x)  satisfying 
sz  =  hAfx;  that  is,  for  all  r  e  / , 
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sz(t)  =  h Kx(t) 

=  lim  h NKx(t) 

iv-x»  (4.3.3) 

=  lim  ZhiiNWxitJi). 

N~*om  i=i 

It  follows  that  Z(X)  =  (X,  sz)b(kx)  is  §*ven  by 

Z(X)  =  h(X) 

=  lim  ZhiiNVCdi). 
i= 1 

Of  course.  Equation  (4.3.3)  is  just  a  generalization  of  the  well-known  Wiener-Hopf  equation. 

The  foregoing  discussion  is  a  restatement  of  classical  L2-estimation  results  in  RKHS 
terms,  but  now  consider  the  problem  in  which  there  is  uncertainty  in  the  structure  of  Kx  or 
sz.  In  particular,  suppose  we  know  only  that  the  pair  (szJCx)  belongs  to  some  admissible 
uncertainty  class  U .  Recall  that,  for  any  pair  (sJC)e  IF,  we  require  that  s  e  H(K).  Since 
we  have  already  seen  that,  in  general,  we  must  have  sz  6  H  (K  x),  this  restriction  on  the 
structure  of  U  is  quite  natural  for  this  problem.  Now,  since  Kx  and  sz  are  no  longer 

A  A 

assumed  to  be  known  precisely,  we  cannot  expect  to  find  the  estimate  Z(X)  e  L\X)  that 
solves  (4.3.1).  Instead,  we  look  for  a  filter  hR  e  F(K(U))  that  satisfies 

sup  M  (hR;  (s  JC))  =  inf  sup  M(h;  (sJC)),  (4  3  4) 

(sJCfeU  heF(K(U))  (sJ()eU  '  ’ 

where  A#  (h;  (sj())  is  the  MSE  associated  with  estimating  Z  by  h(X)  if  Kx  =  K  and  sz  =  s\ 


that  is. 
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M (h;  (sJC))k  E{[Z  -  h(X)]2} 


=  E{[Z  -  lim  ^hi(N)X(ti)]2} 


N 


i=l 


=  E{Z2}  -2 
=  E{Z2}  -  2 


lim  E/iiCAOEtfC^JZ} 


t=i 


+  lim  ^hi(N)h.(N)E[X(ti)X(tj)} 
N~*~  »./=! 


lim  lihi(N)s(ti) 


N 


i=l 


+  lim  ZhtmhjQ/yKitjJi) 
N~>~  i,j=l 


=  o|  -  2  (s ,  h)  +  (hAT,  h). 


Notice  that  we  are  implicitiy  assuming  that  oj  =  E[Z2}  does  not  depend  on  the  actual  struc¬ 
ture  of  /if  x  or  sz  -  it  is  assumed  fixed  but  not  necessarily  known.  This  is  usually  a  reasonable 
requirement  on  the  structure  of  the  uncertainty  in  the  problem  (see  [52]  and  [66]  for  some 
examples),  and,  in  any  case,  it  is  equivalent  to  solving  a  "normalized"  problem  in  which  the 
performance  criterion  is  MSE/o|,  and  U  consists  of  pairs  (sJC)  corresponding  to  the  possible 
values  of  Csz/a|,  ATx/Oz). 

A  filter  hR  €  F(K  (U))  satisfying  (4.3.4)  is  called  a  robust  filter  for  the  game 
(F(K  (U)),U  M)-  In  order  to  find  a  robust  filter,  we  search  for  a  saddle  point  for 
(F  (K(U)),U  M),  which  is  defined  to  be  a  pair  (hR;  (s^JC^))  e  F(K(U))y.U  that  satisfies 

M (hR;  (s  JK))  <  M (hR;  (s J)  Z  M (lv,  (sLJCO)  v  h  s  F(IK(U)),  (s,K)  s  U. 

(4.3.5) 

If  (hR;  (s l,K0)  satisfies  (4.3.5),  then  it  is  easy  to  see  that  (see  [4],  §2.3.1) 


inf  sup  M (h;  (s,K))  =  sup  inf 
h«=F(Jf(ir))  ( s,K)eU  ( s,tC)etf  heF(K(U)) 


M  (h;  (s  JC))  —  M  (hR;  (*L,KL)). 


In  particular,  hR  satisfies  (4.3.4).  If  we  define 
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inf  M(h; 
h  eF(K(U)) 

then  (4.3.5)  clearly  implies  that 

that  is,  (iLXi)  is  least  favorable  for  {F  (JK(U)),U  M)-  We  consider  some  of  the  properties 
of  M  and  M* . 

Lemma  4.3.1:  M  is  concave  on  U  for  fixed  h  €  F(JK  (U)). 

Proof:  Let  (siyKt)  e  U  for  i  =  0,  1  and  (ja^o)  =  ((l-a)r0  +  oejlf  (l-a)AT0  +  a K t)  for 
0  £  a  <  1.  Then,  for  fixed  h  €  F(K{U)), 

M  (h;  (s  a))  =  o|  -2  (sa,  h)  +  (h Ka,  h) 


=  (1-a)  (a|  -  2  is q,  h0)  +  [hK0,  h)j 


+  a[c|-2(.s1,  h) +  (!»*,,  h>] 

=  (l-<xW(h;  (s0JC0))  +  aM(h;  (s^O).  ■ 

Lemma  4.3.2:  M*  is  concave  on  U . 

Proof:  Let  (saJC a)  be  as  before.  Then 
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M*  (saJKo)  =  inf  M (h;  (s^J) 

heF(K(U)) 

>  inf  f(l-a)M(h;  (s0JC 0))  +  aM(h;  (SiM))  1 

h eF(K(U))  l  J 

>  (1-a)  f  inf  M(h;  (j0^o))1 +  af  inf  W(h;  (j^)) 

(heF(tf(ff))  J  lheF(#T(V)) 

=  (l-a)M *(j0^o)  +  ctM*(slJ(1).  U 

Lemma  4J.3:  For  any  (j0^T0)  e  U , 

M  (s  oS  o)  =  °z  ~  Vs  o»  5  O/ff  (*■„)• 

Further,  for  any  h  €  F(JK(U)),  we  have 

Mfli;  {sqJCq))  =  M\sqJ(q) 


if  and  only  if  hAT0  =  s0. 

Proof:  Since  50  €  H(K0),  there  exists  h0  €  F (K 0))  such  that  h^g  =  j0.  For  any 
h  €  F(K{U))  <zF{Kq),  we  have 

M(h;  (s q,K q))  =  a|  -  2  (50.  h)  +  (h^0,  h) 

=  °Z  ~  2  (ho^0,  h)  +  (h/T0,  h) 

=  aZ  ~  2  (h^Q,  hA'0)Jff(^0)  +  (hA'o,  h^o)^^ 

=  a|  +  \(h-h0)^  0,  (h— h0)^T 0)jf  (/c0)  -  (M^o>  ho^  o)n  (at0) 

=  a|  +  ((h-h0)Af0,  (h-h0)/fo)a(X-g)  -  (s  s0)a(Koy 

Hence, 


M  (h;  (J0^o))  ^  az  ~  (5 0’  ^O^CATo)’ 


with  equality  if  and  only  if  hK0  =  h0K0  =  s0.  Further,  since  h0  €  F(K0),  there  exists  a 
sequence  {hjv}/v=i  <=  FUK(U))  of  finite  filters  such  that 
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^Jim  h$Ko  =  ®*o^”o  =  5o» 

and  it  follows  that 

lim  M(h$;  (sQJC^))  =  c|  -  2  f  lim  {sQ,  h$)l  +  lim  (hjv^o*  ^n) 
iV->—  l/v-*-  J 

=  o|  -  2  j^lim^  (i0-  hjv^o)*(Jt0)j  +  ^  $nk o»  W^0)h(a:o) 

=  °Z  “  2  (sQ,  s£a(Ko)  +{s  q,  S  tyM(Ko) 

=  °z  “v5  o>  so/a(Kci- 

This  proves  the  lemma.  ■ 

We  can  now  give  necessary  and  sufficient  conditions  for  a  pair  (hR;  (5  ^,K  l))  to  be  a  sad¬ 
dle  point  for  (F  (JK(U)),U  M)-  These  conditions  are  very  reminiscent  of  those  given  by 
Poor  in  [53],  Lemma  1,  relating  to  saddle  point  solutions  for  the  robust  matched  filtering 
problem.  The  similarities  between  these  two  problems  will  be  explored  more  fully  in  the  next 
section. 

Theorem  4J.4:  (hR;  (Jl^l))  e  F(JK(U))  x  17  is  a  saddle  point  for  (F {IK (U)),U M) 
if  and  only  if  h  RK  l  =  s  l,  and 

2(j,hR;-(jL,  hR)-;hRK,h^'2  0.  V(,X)€t7.  (4.3.6) 

Proof:  Recall  that  (hR;  (sl^l))  is  a  saddle  P°im  for  {F(K{U))tU  M)  if  and  only  if 

M  (hR;  (Jl^Tl))  <  Affli;  (j^l)).  v  h  e  F{K{U)),  (4.3.7) 

and 

M(hR;  (sJC))  <.  M (hR;  v  (sj()  e  U.  (4.3.8) 

It  follows  immediately  from  Lemma  4.3.3  that  (4.3.7)  is  satisfied  if  and  only  if  hRAT L  =  if  so 
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the  theorem  will  be  proven  if  we  can  establish  that,  given  hR^TL  =  ^l>  (4.3.8)  holds  if  and 
only  if  (4.3.6)  holds.  To  this  end,  notice  that,  since  M  is  concave  on  U  for  fixed 
h  e  F{K(U)),  (4.3.8)  holds  if  and  only  if,  for  each  (s£)  e  U, 

lim  —  [m(Hr;  (SaJCJ)  -  M (hR;  (sLJCO)}  £  0, 

a-*0  Ct  v.  J 

where  ( satKa)  =  ((l-a)sL  +  as ,  0~a.)KL  +  aK).  But  if  hR£L  =  s l,  then 

\s  L»  s  UB(KO  -  \s  L*  hR)  =  \hR^L»  hRr 

and,  by  Lemma  4.3.3, 

M  (hR;  (sLJCi))  =  a|  -  [sL,  s£a(Kj. 

Therefore, 

fim  -±-  [M(hR;  (saJCa))  —  M (hR;  (j^l))]  =  Hm  ±  [-2a  [s ,  hR)  -  2(l-a)  [sL,  hR) 

+  \hR*a>  hR)  +  (hR^L’  Hr)] 

=  lim  —  [-2a  (s,  hR)  -  2(l-a)  [sL,  hR) 

+  a  (h rK  ,  hR)  +  (2-a)  (hR^f  hR)j 

=  lim  j-  [-2a  (s ,  hR)  +  a  ' s  L,  hR)  +  a  (h RK ,  hR) 
a-»0  a  l  J 

=  -2  (j,  hR)  +  is L,  hRx  +  (hRAT,  hRx 

Hence,  if  hR^rL  =  JL,  then  (4.3.8)  holds  if  and  only  if  (4.3.6)  holds,  and  the  theorem  fol¬ 
lows.  ■ 

Theorem  4.3.4  gives  a  complete  characterization  of  saddle  point  solutions  for 
{F  (IK  {U  )),U  M  )•  The  next  theorem  and  its  corollary  give  conditions  under  which  the 
existence  of  a  saddle  point  is  equivalent  to  the  existence  of  a  least  favorable  pair  ($l.^l) 
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maximizing  M*  on  U .  Since  M*  is  concave  on  U ,  the  search  for  a  least  favorable  pair 
often  reduces  to  a  straightforward  convex  programming  problem.  The  proof  of  Theorem  4.3.5 
is  given  in  Appendix  B. 

Theorem  4.3.5:  Let  (s^l)  e  ^  and  let  hR  e  F(KO  satisfy  hRtfL  =  jl.  Suppose  that 
Kl  dominates  K(U);  that  is,  for  all  AT  6  BC(U),  there  exists  C  >  0  (depending  on  AT)  such 
that  K  c  CKl  (i.e.,  (CAfL  -  AT)  is  nonnegative  definite  on  I).  Then  hR  e  F{IK{U))  and  the 
condition 

Vs L»  sL/B0 <0  ^  \s '  s)r(K)’  v  (sJC)  e  U,  (4.3.9) 

holds  if  and  only  if  (4.3.6)  holds;  that  is,  if  and  only  if 

2  (* .  Hr)  -  L.  hR)  -  (hRAT,  Hr)  £  0,  V  (SJC)  e  U. 

Corollary  4.3.6:  Suppose  that  ( s^JCO  is  a  least  favorable  pair  for  (IF (K {U)),U M)  and 
that  Kl  dominates  K(U).  Let  hR  e  F{Ki)  be  such  that  hRAfL  =  sL.  Then  (hR;  (sl»^l))  is 
a  saddle  point  for  ( F(K(U)),U  M)- 

Proof:  If  <s  LJC0  *s  a  least  favorable  pair,  then 

\s L,  s £(s,s )*(*),  V  (sJC)  e  U. 

Since  KL  dominates  K{U ),  it  follows  from  Theorem  4.3.5  that  hR  e  F  (K  (VJ))  and 
2(*.Iir)-£l.  hR)-(hRAr,  hR)^0,  V  (SJC)  e  U. 

The  result  follows  by  applying  Theorem  4.3.4.  ■ 

Remark  4.3.7:  Suppose  that  the  index  set  I  is  itself  a  Hilbert  space  H0.  Then  the 
covariance  functions  K  e  K(U)  all  correspond  to  bounded  linear  operators  on  H  0,  and  a 
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sufficient  condition  for  KL  to  dominate  K(U)  is  that  the  operator  KL  be  invertible.  To  see 
this,  let  AT  e  K( U)  with  corresponding  operator  K  and/  e  B0.  Then 

If.  (C  Kl  -  K V)m,’!f.  kl<CI  -  K^KKC'^Kfr)^ 

=  (K ?/,  (Cl  -  K^KKC^Kf/)^ 

=  C||Kfr||£,  -  (K  ?/,  (Kj^KKC'^K,*/}*, 
21|Kfrlli„[C  -  HKC,/4KKClAll  ]■ 

Clearly,  if  C  is  chosen  to  satisfy  C  >  ||K£%KK£V*||,  then  the  operator  (CKL  -  K)  is  positive 
and  K  c  CKL. 

It  follows  that,  if  (j l^l)  Is  a  least  favorable  pair  for  (F (K (U)),U M)  and  the  operator 
Kl  is  invertible,  the  pair  (K^1^;  (s LJC  J)  is  a  saddle  point  for  (F (JK(U)),U M)-  Note  that, 
in  this  case,  the  filter  hR  consists  of  a  single  element  hR  =  Kl'sl  e  H0.  The  corresponding 
random  variable  is  hR(X)  =  X(/i  R). 


In  the  case  when  the  covariance  structure  of  the  process  X  is  assumed  to  be  known,  so 
that  the  only  uncertainty  is  in  the  structure  of  the  "signal"  sz ,  the  existence  of  a  robust  filter  is 
guaranteed,  as  the  following  corollary  shows. 

Corollary  4.3.8:  Suppose  U  has  the  form 

U  =  {( sJCq ):  s  e  5  c  H(K0),  S  convex) 

for  some  fixed  covariance  function  Kq.  Then  there  exists  a  robust  filter  hR  for  the  game 
(F  {K0),U  M)-  Further,  hR  satisfies  hR/C0  =  rL,  where  is  the  unique  element  of  S  (the 
closure  of  S  in  H(K  0))  with  minimum  norm;  that  is. 


80 


V*L*  SUR(,KA  ^  Vs*  5/»(*e)’  V  5L  e  S. 


(4.3.10) 


Proof:  Let  W  =  {(sJCQ):  s  e  5).  Since  5  is  a  closed,  convex  subset  of  U(K0),  there 
exists  a  unique  element  sLe  S  satisfying  (4.3.10)  (see  [31],  §3.12).  Since  sLe  U (K0),  we 
can  find  hR  satisfying  hRAT0  =  sL,  and,  by  Corollary  4.3.6,  (hR;  (sL,K0))  is  a  saddle  point  cor 
( F(K0),UM ).  Since  M*  (s  JK0)  -  of  -{s,  s)a(Kj  is  a  continuous  function  on  S,  it  follows 
that 


C* 


M*  (sJCq) 


=  sup 
(sJC'fcff 


Hence,  following  [70],  §H.D,  we  have 


M  (hR;  (s  L JC 0))  =  sup  M  (hR;  ( s  JC 0» 

£  sup  M(hR;(sJT0)) 

(s.K  ofeu 

>  inf  sup  M(h ;  (j/0)) 

heF(K0)  U'Koku 

£  sup  inf  M(h;  (s,KQ)) 

UJC< feU  heF(ATo) 

=  sup  M*(sJ(  o) 

(*j:a5ea 

=  sup  M*(sJC0) 

(sJC„}eU 

-  =  sup  inf  M  (h;  {s  ,K  0)) 

{s,K0)eU  he^(ATo) 

=  M  (hR;  ( s  l JC  0)). 


It  follows  that  all  of  the  inequalities  above  can  be  replaced  with  equalities,  and  we  get 


sup  M (hR;  (s,Kq))  =  inf  sup  M (h;  (5  JC 0)), 
(sjeJeU  heF(Ko)(s.K0kO 


so  that  hR  is  a  robust  filter  for  (F(K0),U  M)-  ■ 
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If  the  covariance  structure  of  X  is  not  precisely  known,  considerably  more  structure  is 
needed  to  guarantee  the  existence  of  a  robust  filter.  The  following  corollary  gives  one  set  of 
conditions  sufficient  to  guarantee  the  existence  of  a  saddle  point  for  (F  {IK  (U)),U  M)- 

Corollary  4.3.9:  Suppose  that  /  is  a  Hilbert  space  H0,  and  let  us  regard  K(U)  as  a  set 
of  bounded,  linear,  self-adjoint,  positive  operators  on  H0.  Suppose  further  that 
K(U)  c  HS(H0),  the  Hilbert  space  of  Hilbert-Schmidt  operators  on  U0  with  the  Hilbert- 
Schmidt  norm  IHIhs(H0)-  can  *en  regard  17  as  a  subset  of  B0  x  HS(I70),  which  becomes 
a  Hilbert  space  under  the  norm 

ll(-,*)ll2  =  Hi,  + 


if 


(i)  for  every  Kq,!^  e  K(W)  there  exist  constants  0  <  c  <  C  <  °°  (depending  on  K0 

and  such  that  c  Kq  c  <c  C  Kq  (i.e.,  the  operators  (C  K0  -  Kt)  and 

(Kj  -  cKo)  are  both  positive),  and 

(ii)  <7  is  a  closed,  bounded  subset  of  £F0  x  HS(J70)» 


then  there  exists  a  saddle  point  for  (F  (K(B)),U M)- 

Proof:  For  the  case  under  consideration  (/  =  H0),  it  is  easy  to  see  that  a  finite  filter  h 
corresponds  to  a  function  h  s  £f0,  and,  for  any  (s  ,K)  s  U , 

,  h;  = 


lu 


hK,  h)  =  (A,  K h)ao, 


and 
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M(h;  (s, K))  =  o|  -  2  (h,  s)Bo  +  (h,  K h)Bo. 

Clearly,  for  fixed  h,  the  function  A/(h;  (•,  *))  can  be  extended  to  a  continuous,  concave 
function  on  all  of  H 0  x  HS(H0).  By  considering  the  proof  of  Lemma  4.3.3,  it  is  easy  to  see 
that,  for  all  (r,K)  e  U, 

M'(s, K)g  inf  M  (h;  (r  ,K)) 

heF(K(U)) 

=  inf  M(ii;(r,K)). 

UF(K(U)) 

Hence,  M *  also  can  be  extended  to  all  of  H0  x  HS(H0)  as  the  infimum  of  a  class  of  continu¬ 
ous,  concave  functions.  It  follows  that  M*  (so  extended)  is  concave  (see  the  proof  of  Lemma 
4.3.2)  and  upper  semicontinuous  (see  [15],  Proposition  7.11).  By  hypothesis  (ii),  U  is  a 
closed,  bounded,  convex  subset  of  the  Hilbert  space  H0  x  HS(H0),  and  it  follows  from  [4], 
Theorem  2.1.2,  that  there  exists  (sl,Kl)  e  U  such  that 

M  *  (s  l,Kl)  =  sup  M*  (s  ,K); 

that  is,  (j^Kl)  is  a  least  favorable  pair  for  (F  (K(U)),U  M)-  By  hypothesis  (i),  KL  dom¬ 
inates  K{U),  and  it  follows  from  Corollary  4.3.6  that  (hR;  (sl,Kl))  is  a  saddle  point  for 
{F  (K(UF))tU  M)  for  any  hR  6  F(K(U))  satisfying  hRKL  =  rL.  ■ 

We  end  this  section  with  an  example  illustrating  the  material  discussed  above. 

Example  4.3. 10:  Let  I  =  L\[a,b]),  the  space  of  square-integrable,  real-valued  functions 
on  the  interval  [ajj],  where  -«•  <  a  <  b  <  <».  Let  kQ  be  a  known,  real-valued,  continuous 
covariance  function  on  [ a,b ]  and  let  Kq  be  the  operator  on  L2([a,b])  generated  by  kQ;  that  is, 
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(Ko f  )(•)  k  jf  (x)k0(-,x)dx,  V  /  6  L\a  ,b  ]). 

a 

Assume  that  Kq  is  strictly  positive.  It  follows  (see,  for  example,  [41],  §6.10)  that  K0  is  a 
compact  operator  on  L\[aM),  and,  if  we  let  {en}“=\  be  the  set  of  orthonormal  eigenfunc¬ 
tions  of  Ko  with  associated  (positive)  eigenvalues  ^  X2  ^  •••.  then  {e„ }  ~=1  spans 
L2([aJ>])-  If  we  regard  B(Kq)  as  a  subset  of  L2([a  ,b]),  then  it  follows  (see  [48],  §9)  that  a 
function  s  e  L2([a,b ])  is  a  member  of  H (Kq)  if  and  only  if 


n=l 


(4.3.11) 


where 


s„  =  js(x)en(x)dx. 


Now,  suppose  that  o|  >  0,  j0  e  U(Kq),  and  e  >  0  are  given  such  that  IU„.Jj(Ko)  <  of 

o* 

and  e  <  X  (s«°)2>  where 


sn°  =  js0(x)e„(x)dx. 


Define  the  set  S  c  H  (Kg)  by 
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S  4 


5  e  Jf(Ko):  ||s  IIar(Ko)  ^  oZ2»  and  J[j(x)  -  s0{x)]2dx  <  e 


•»  ^ 


4  e  L2([a  £]):  2r  *  and  £(*„  -s„°)2  <1  e 

n- 1  n=l 


(4.3.12) 


We  seek  a  saddle  point  for  the  game  (F(K^,UM),  where  B  =  S  x  (Ko).  We  proceed  by 
searching  for  the  least  favorable  signal;  that  is,  the  signal  s  L  e  S  that  satisfies 


11^  l11  Jr  (Ko)  =  “j* 


(4.3.13) 


It  follows  from  (4.3.11)  and  (4.3.12)  that  solving  (4.3.13)  is  equivalent  to  minimizing  £  — 

n=l  Kn 

subject  to  the  constraint  £  ( sn  -  sn°)2  <>  e.  A  straightforward  Lagrange  multiplier  calculation 

n=\ 

yields  the  candidate  sL  6  L2([aJ)])  defined  by 


SL  =  XSnen  =  Z 
n=l  n=l 

where  o  >  0  is  the  unique  positive  solution  to 

£(i„°-j„L)2=  £ 

n=l  n=l 

Notice  that  (4.3.14)  and  (4.3.15)  imply  that 


+  1 


en. 


c\n  +  1 


=  e. 


M  (S  M 

«=1  Kn  *=1 


n—\ 


<*K  +  1 


(4.3.14) 


(4.3.15) 


=  Xxa2e  <  =», 


(4.3.16) 


so  that  jl  e  B(Kq). 
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To  show  that  jl  is  indeed  the  least  favorable  signal,  we  use  the  fact  that  Sl  satisfies 
(4.3.13)  if  and  only  if  (see  [31],  §3.12,  or  apply  Theorem  4.3.5) 

\S  ~  SL’ s  L/B  (Ko)  ~  °*  V  S  e  S. 

It  follows  from  (4.3.11),  (4.3.14),  and  (4.3.15)  that 

(s  -  sL,  s i)a(Ko)  =\s  -  •Jo*  si}m(Ko)  +  (^o  ~  si}*(k<>) 

=  \S  -  Sq,S [^(Ko)  +  It" 

n=l  Kn 


S° 

o\ns? 

l°*« + 1  j 

cXn  +  1  _ 

=  .'*  - 


\s  -  50»  5UJf(Ko)  +  OE- 


Further,  it  follows  from  (4.3.11),  (4.3.12),  (4.3.16),  and  the  Schwarz  inequality  that,  for  all 
s  e  S, 


I  \s  ~  sO>  ■yL).ff(Ko)1  “ 


(sn  -  Sn°)SnL 


n=l  Kn 


09 

Vi 

oo 

rs  l  i 

2 

< 

I 

X. 

n=l 

/1=1 

L^n  J 

Vi 


<,  OE. 


Hence, 

v s  -  SL’  V  5  e  5, 

and  it  follows  that  jl  is  the  least  favorable  signal. 

Now,  define  h R  €  L2([a,b ])  by 
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~  Zyen- 

n=l  Kn 


(4.3.17) 


Then, 

R  =  2  T  =  X  sn*en  =  5 L» 

n=I  A*  n=l 

and  it  follows  from  Corollary  4.3.6  that  (h  R;  (sl>Ko))  is  a  saddle  point  for  (IF  (K0)M  M)- 
Note  that,  here  again,  the  filter  hR  consists  of  a  single  element  /zR.  The  corresponding  ran¬ 
dom  variable  is 

b 

hR(X)  =  X(hK)  =  jhR(x)X(x)dx. 


We  now  turn  to  the  problem  of  robust  matched  filtering.  As  we  shall  see,  the  structure 
of  this  problem  is  essentially  the  same  as  that  of  robust  L2  estimation. 


4.4.  Robust  Matched  Filtering 

Let  X&[X(t)\t  e  1}  be  an  observed  process.  Consider  the  following  simple 
hypothesis  testing  problem: 


Hq:  X  has  mean  zero  and  covariance  function  Xx 
versus  (4.4.1) 

Hj:  X  has  mean  function  m  and  covariance  function  ATX, 

where  Xx  is  a  known  covariance  function  and  0  *  m  e  H(Xx)  is  a  known  deterministic  sig¬ 
nal.  This  corresponds,  of  course,  to  the  problem  of  detecting  the  signal  m  in  the  presence  of 
additive,  zero-mean  noise  with  covariance  function  Afx.  The  condition  m  e  H  (K  x) 
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guarantees  that  Problem  (4.4.1)  is  well-behaved  and,  when  X  is  Gaussian,  is  related  to  the 
nonsingularity  of  (4.4.1),  as  discussed  in  Chapter  2.  We  shall  have  more  to  say  about  this 
condition  at  a  later  time. 

Problem  (4.4.1)  is  usually  decided  by  using  a  linear  detector,  that  is,  a  detector  in  which 
the  test  statistic  h(X)  is  of  the  form 

h(X)=  lim  ZhiiNVUti), 

N^~  ;=1 

where,  for  each  N ,  the  sequence  {/t,(A,)},“i  czR  has  only  finitely  many  nonzero  elements, 
and  convergence  is  the  mean-square  sense  under  both  hypotheses.  Clearly,  given  any  such 
statistic,  there  is  a  corresponding  filter  h  e  F  (K x).  Furthermore,  since  m  e  H(KX),  it  is 
easy  to  see  that,  given  any  h  e  F  (K x),  the  corresponding  statistic  h(X)  is  well-defined  under 
both  hypotheses. 

The  signal-to-noise  ratio  (SNR)  for  Problem  (4.4.1)  corresponding  to  the  filter 
h  €  F(Ky)  with  associated  test  statistic  h(X)  is  given  by1 


'Here  and  elsewhere,  the  quantity  is  defined  to  be  0. 
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Hi(h(X)} 

A(h;  [mJCx))  k  Var(){h(X)j 

'  "12 

lim  'Zthi(N)m(ti) 

N~*°  1=1 

—  J 
lim  YlhimhjmKxitrti) 

N-*~  i= 1 


(hX'x,  hX'x)fl(Arx) 


(4.4.2) 


A  matched  filter  for  Problem  (4.4.1)  is  any  filter  h*  6  F(KX)  that  satisfies 

A(h* ;  (m  JC x))  =  sup  A(h;  (mJCx)).  (4.4.3) 

heFfAT*) 

It  follows  immediately  from  (4.4.2)  and  the  Schwarz  inequality  that  h*  satisfies  (4.4.3)  if  and 
only  if  h*  A'x  =  cm,  for  some  constant  c  *  0,  and  the  maximum  SNR  for  Problem  (4.4.1)  is 
given  by 

A(h*;  ( \mJCx ))  =  INIIa(iCx)- 

Now,  suppose  that  the  pair  (m  JC  x)  is  known  only  to  belong  to  some  admissible  uncer¬ 
tainty  class  U\  that  is,  we  now  consider  the  composite  hypothesis  testing  problem: 
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Hq:  X  has  mean  zero  and  covariance  function  K, 
versus  (4.4.4) 

Hj:  X  has  mean  function  s  and  covariance  function  K , 


where  (sJC)  e  U }  In  order  to  decide  Problem  (4.4.4),  we  are  interested  in  using  a  linear 
detector  incorporating  a  test  statistic  hR(X)  corresponding  to  a  filter  hR  €  F(K(U))  that 
satisfies 


inf  A(hR;  Cr,X))  =  sup  inf  A(h;  (sJC)). 
(sJOeU  heF(K(U))  (sJC)eU 


(4.4.5) 


We  will  refer  to  a  filter  satisfying  (4.4.5)  as  a  robust  matched  filter  for  the  game 
(F(K  (U)),U  A)-  In  order  to  find  a  robust  matched  filter,  we  again  search  for  a  saddle  point 
(hR;  ( sLJ(0 )  €  F(K(U))x  U  satisfying 


A(h;  (j^l))  £  A(hR;  (sLJ<0)  *  A(hR;  (sJC)), 

If  we  define 


V  h  e  F(K(U)\  (s,K)  e  U. 

(4.4.6) 


A  *(sJK)&  sup  A(h  ;(sX)), 

teF(K(U)) 

then,  as  before,  (4.4.6)  implies 

aVjl^^^A*^^),  V(sJOeU, 

so  that  (sLJCO  is  least  favorable  for  (F  (K(U)),U  A)-  We  have  the  following  properties  for 
A  and  A* . 


2We  assume  throughout  this  section  that,  for  any  pair  (sK)  e  U ,  s  *0. 
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Lemma  4.4.1:  A  is  convex  on  U  for  fixed  h  e  F(K ( U )). 

Proof:  Let  (st  JCt )  e  U  for  i  =  0,1,  and  define  (saJC «*)  in  the  usual  manner  for 
0  £  a  £  1.  If  (hATj,  h)  =  0  for  i  =  0  or  i  =  1,  then  it  is  easy  to  see  that 

A(h;  (saJCa))  =  (l-a)A(h;  (s0JC0))  +  aA(h; 

If  (HAT,-,  h)  >  0  for  t  =  0  and  i  -  1,  then,  letting 

a  (hATlt  h) 

P  "  (htf  a,  h)  ’ 


we  have  p  e  [0,1]  and 


A(h;  (SaJCa))  =[hKa,  h) 


=  (l-a)A(h;  (50^0))  + 


This  proves  the  lemma.  ■ 

Lemma  4.4.2:  A*  is  convex  on  U . 

Proof:  See  proof  of  Lemma  4.3.2. 

Lemma  4.4.3:  For  any  (SqJCq)  e  U , 

A*  (s  or^o)  =  (s  o»  s  ola  (K0y 
Further,  for  any  h  6  F(JK{U)),  we  have 

A(h;  (sqJCq))  =  A  (sqJCq) 
if  and  only  if  h/if0  =  cs0  for  some  c  *  0. 
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Proof:  By  the  Schwarz  inequality,  we  have 


A(h;  (SqJC o))  = 


r 


[hK0,  hK 


T - 

O/HCATo) 


^  Vs  0>  5  0/i 


with  equality  if  and  only  if  hAf0  =  cy0,  c  *  0.  Further,  since  50  e  B(K0),  there  exists  a 
sequence  {h$  }£*=1  of  finite  filters  such  that 


JV 


lim  —  Sq, 


and  it  follows  that 


lim  A(hg;  (sqJCq))  =  lim 
/V-+~  iV-+-> 


(|*N*0*  ^V^o)iP(Af0) 


=  (s  o,  So,B(Ko)- 


This  proves  the  lemma.  ■ 

The  following  theorem  gives  necessary  and  sufficient  conditions  for  a  pair  (hR;  {s  ^,K  l)) 
to  be  a  saddle  point  for  <JF  (K  (U)),U  ,A).  It  should  be  noted  that,  except  for  the  addition  of 
a  scale  factor  on  the  robust  filter  hR,  these  conditions  are  identical  to  those  given  in  Theorem 
4.3.4  for  the  robust  L2-estimation  problem. 

Theorem  4.4.4:  (hR;  {sLJC  J)  e  F{JK(U))  x  U  is  a  saddle  point  for  {F{IK{U)),U, A) 
if  and  only  if  hR/fL  —  cs  l  for  some  c  *  0,  and 


2(5,h'R)-(jL,  h'R)  -  (h'RK ,  h'R)  >  0, 


*  (sJC)  6  U, 


(4.4.7) 
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where  h'R  =  — hR. 

c 


Proof:  It  follows  from  (4.4.6)  that  (hR;  (5  l))  is  a  saddle  point  for  (F  (IK  {U)),U , A)  if 

and  only  if 


A(h;  (sL*0)  *  A(hR;  (sLXO)>  v  h  e  F(K( 17)),  (4.4.8) 


and 


A(hR;  (5l^l))  £  A(hR;  (, s  JC )),  V  (sJC)  e  U.  (4.4.9) 

Lemma  4.4.3  implies  that  (4.4.8)  is  satisfied  if  and  only  if  hR.KTL  =  csL  for  c  *  0,  so  we  need 
to  establish  that,  given  hR/ifL  =  csL,  (4.4.9)  holds  if  and  only  if  (4.4.7)  holds.  Since  A  is  con¬ 
vex  on  U  for  fixed  h  e  F(K(U)),  (4.4.7)  holds  if  and  only  if,  for  every  (sJC)  e  U , 

lim  [a(1ir;  (s^J)  -  A(hR;  (sL£0)}  *  0 

a-»0  L  J 

where  Csa»^a)  is  defined  in  the  usual  way.  But  if  hR/CL  =  csL,  then 

A(hR;  (sLJC  l))  =  (s  L,  s  i)jf  j  =  ~  L>  ~  (h RfC  L,  hR), 


and 


fA(hR;  {saKa))  -  A(hR;  (sl^l))1  =  lim  - 
o— »0  <X  v.  J  a-*0  a 


fca’ hR/)  1  , 


\hR/^a>  hR/  C 


V  -  —  Vs  L>  hR/ 


=  lim  — 
a-*0  a 


a2  ((* .  Hr)]2  +  2a(l-a)  is ,  hg)  is L,  Hr)  +  (1-a)2  [(sL,  hR)]2  }  ^ 


\hR^a.  hit) 


~  "7  \SL>  hR/' 


"  c  \S'  hfb  ~  c  t?L’  *lR/  +  (^  ca  j^L*  hR)  “  “  \^R^a-  ^r)) 

-  7  If-  "r)  -  7  !fu  *>R>  +  “3  73J  [“  ("R«L.  i>r)  -  a  (hR*r.  I>r)] 
=  T  ■  hR>  ~  T  (SL.  "r)  -  4  (hR*.  hR> 


=  2[s,  h'R)  -  is  L,  h'8;-(h'RAr,  h-j). 


Hence,  if  hR/SrL  -  csL,  (4.4.9)  holds  if  and  only  if  (4.4.7)  holds,  and  the  theorem  follows.  ■ 
We  also  have  the  following  corollary,  which  is  analogous  to  Corollary  4.3.6. 

Corollary  4.4J5:  Suppose  that  (sLyKi)  is  a  least  favorable  pair  for  (F  {K  (U)),U  A)  and 
Kl  dominates  K{U).  Let  hR  be  a  matched  filter  for  {sLKO>  that  is,  hRATL  =  csL  for  some 
c  *  0.  Then  (hR;  (x^l))  «  a  saddle  point  for  (F  {K  {U)),U  A). 

Proof:  Let  h'R  =  -^-hR  so  that  h'R£L  =  sL.  By  hypothesis,  (s  LK  L)  is  a  least  favorable 
pair,  so,  for  all  (sJC)  e  U, 

A  (sL,K0  =  (sL’  (kl)  -  \5>  5/Jf(JO  =  A  (sK)- 


Since  K L  dominates  IK  (17),  it  follows  from  Theorem  4.3.5  that  h'R  e  IF  {IK  (U))  and 
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2 (j,  h'R) -is L,  h'R) - (H'rAT ,  h'R) ;>  0,  V  (sJOe  U. 


The  result  follows  by  applying  Theorem  4.4.4.  ■ 

At  this  point,  it  is  clear  that  the  characteristics  of  a  saddle  point  solution  to  the  robust 
matched  filtering  problem  are  virtually  identical  to  those  of  a  saddle  point  solution  to  the 
robust  L2-estimation  problem.  Indeed,  if  we  insist  upon  using  normalized  matched  filters  (i.e., 
hRATL  =  sj,  the  characteristics  are  the  same.  This  being  the  case,  we  can  merely  observe 
that,  with  the  exception  of  Corollary  4.3.9  (the  proof  of  which  depends  explicitly  upon  the 
structure  of  the  function  M ),  all  of  the  remarks,  corollaries,  and  examples  following  Corollary 
4.3.6  have  exact  analogs  relating  to  the  robust  matched  filtering  problem.  In  particular,  if  the 
covariance  structure  of  the  observed  process  is  assumed  to  be  known,  so  that  the  only  uncer¬ 
tainty  is  in  the  structure  of  the  signal  m ,  the  existence  of  a  robust  matched  filter  is  guaranteed. 

With  reference  to  the  case  in  which  the  covariance  function  ATX  is  assumed  to  be  known, 
it  is  interesting  to  note  that  the  matched  filter  corresponding  to  any  signal  m  e  H(K\)  will 
be  a  robust  matched  filter  for  any  admissible  uncertainty  class  U  in  which  the  least  favorable 
signal  is  a  multiple  of  m .  This  follows  immediately  from  Corollary  4.4.5  and  holds,  in  partic¬ 
ular,  for  the  class  U  given  by 


U  4 


(S  jKx)'-  11$  “  Kx)  ^  £ 


for  any  0  <  e  <  (*x)-  This  is  a  generalization  of  a  more  familiar  result  regarding 

matched  filters  for  signals  in  white  noise  (see  [25],  §III.A).3 


3 


A  similar  result  has  been  noted  recently  by  Donoho  and  Liu  in  a  different  robustness  framework  [13], 


Remark  4.4.6:  For  the  robust  matched  filtering  problem,  the  requirement  that  s  e  H(K) 
for  all  (sJC)e  U  is  not  as  natural  as  it  is  for  the  L2-estimation  problem.  Although  such  an 
uncertainty  class  does  usually  admit  a  rich  class  of  signal-covariance  pairs,  it  is  sometimes  the 
case  that  the  "natural”  uncertainty  class  contains  pairs  for  which  s  4  H(K).  For  example,  if 
/  is  a  Hilbert  space  H0  and  Kq  is  a  known  covariance  operator  on  H 0,  a  common  uncertainty 
class  corresponding  to  signal  uncertainty  is  of  the  form  S  x  {Kq},  where 

S  4  'S  e  Hq:  ||s  -Sol&oSe 
» 

for  some  nominal  signal  sQ  and  0  <  £  <  ||$ollff0  ^sec  t53]).  If  Kq  is  not  an  invertible  operator, 
such  a  set  will  contain  signals  that  are  not  in  H(Kq). 

Consider  the  problem  of  finding  robust  matched  filters  for  the  case  in  which  the  uncer¬ 
tainty  class  U  contains  pairs  (sJC)  for  which  s  4  H(K).  Naturally,  we  must  restrict  our 
attention  to  the  set  of  filters  F(U)  c  F(IK(U))  that  are  well-defined  for  all  signals  "con¬ 
tained  in"  U.  That  is,  h=  {h/v  };v=i  is  a  member  of  F{U)  if,  for  every  (s  ,K)  e  U, 
h  e  F(K),  and 

(s,h)4  lim  (s ,  hN)  4  lim  Y/hi(N)s(ti) 

N-*~  N  — l=j 

is  well-defined.  If  we  seek  saddle  points  for  this  problem,  then,  as  the  following  lemma 
shows,  under  mild  regularity  conditions,  we  can  identify  all  possible  saddle  points  by  restrict¬ 
ing  attention  to  those  pairs  (j  )  €  U  for  which  s  e  H(K).  The  proof  of  Lemma  4.4.7  is  a 
straightforward  application  of  the  RKHS  Approximation  Lemma,  which  is  given  in 
Appendix  C. 
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Lemma  4.4.7:  Let  /  be  a  separable  metric  space  and  U  a  convex  set  of  pairs  (sJC)  such 
that  if  is  a  continuous  covariance  function  on  /  and  s :  I  — >  JR  is  continuous.  Then 


*|2 

sp*) A(h:  t*  ’*» 4  i«fFo)  r  < 


if  and  only  if  s  e  H(K).  Defining  the  associated  admissible  uncertainty  class  U'  by 


U'  A 


(sJC)  6  U:  s  €  H (K) 


} 


and  assuming  that  U'  is  nonempty,  it  follows  that  any  least  favorable  pair  (s^l)  f°r 
( F(U),UA )  must  satisfy  6  U'.  Hence,  (hR;  (s  l^l))  is  a  saddle  point  for 

(F(U),U  A)  only  if  it  is  also  a  saddle  point  for  (F(K  (U^XU'A)- 


Note  that  the  regularity  conditions  for  Lemma  4.4.7  are  not  very  restrictive.  They  are 
satisfied,  for  example,  if  /  is  a  countable  set  (endowed  with  the  trivial  metric),  or  if  /  is  a 
separable  Hilbert  space  Hq,  and  U  corresponds  to  a  collection  of  pairs  (s  ,K)  with  s  e  H0 
and  K  a  bounded  covariance  operator  on  H0.  Also  note  that  Lemma  4.4.7  does  not  say  that 
if  (hR;  (rL^L))  €  x  U'  is  a  saddle  point  for  (F  [K{H')),U> ,A),  then  it  is  necessarily 

a  saddle  point  for  (F  (U),U  A)-  However,  in  many  problems  of  interest,  this  will  in  fact  be 
true,  as  the  following  lemma  indicates.  The  proof  of  Lemma  4.4.8  is  given  in  Appendix  C. 

Lemma  4.4.8  Let  /  be  a  Hilbert  space  Hq  and  let  U  =  S  x  IK ,  where  S  c  H0  is  con¬ 
vex  with  a  nonempty  interior  and  K  is  a  convex  class  of  covariance  operators  on  H  0.  Let 
V  A  {(s,K)  €  U:  s  €  H(K)}.  If  there  exists  fcR  e  Hq  and  (sl>Kl)  e  such  that  KL  is 
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strictly  positive  and  (h  R;  (sl,Kl))  is  a  saddle  point  for  (F(K(U')\ IT, A),  then  (h  R;  (sL,KL)) 
is  a  saddle  point  for  (F  (U),U  A)- 

We  close  this  section  with  a  simple  example. 


Example  4.4.9:  As  in  Example  4.3.10,  let  /  =  L2([a  ,b  ]),  be  a  continuous  covariance 
function  on  [aj>]  with  associated  strictly  positive  covariance  operator  Kq,  and  s0  e  U(K0). 
Let  U  =  5  x  {Kg},  where 


5  4 


s  e  L2([a,b]):  J[s(t)  -  s0(x)]2dx  <>  e 


b 

for  some  0  <  e  <  Jsq  (x)d  x.  As  discussed  above,  we  define  U'  as 

a 


u'  £ 


j (s >Kq)  €  U:s  e  IT (Ko)k 


and  it  is  obvious  that  U’  -  S'  x  { K<j},  where 


S'4 


r  e  H  (Kg):  -  j0(t)]2dx  £  £ 


Proceeding  exactly  as  in  Example  4.3.10,  we  find  that  ( hR ;  (rL,Ko))  is  a  saddle  point  for 
(F(Ko),l7',A),  where  sL  and  hR  are  defined  by  (4.3.14)  and  (4.3.17),  respectively.  It  follows 
from  Lemma 4.4.8  that  ( hR ;  (5L,Kg))  is  also  a  saddle  point  for  (F(U\U  A)4 

“'This  generalizes  an  example  given  in  [53],  in  which  it  was  shown  that  /iR  is  a  robust  matched  titter  for  the 
game  (L  ([aJ>]),U  A).  In  our  terminology,  this  corresponds  to  restricting  the  class  of  allowable  filters  to  in¬ 
clude  only  finite  filters. 
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4.5.  Robust  Quadratic  Detection 

Throughout  this  section,  we  assume  that  the  observed  process  {X(f);r  e  /)  is 
Gaussian  with  mean  zero.  Consider  the  hypothesis  testing  problem: 

Hq.-  X  has  covariance  function  Ky 

versus  (4.5.1) 

Hp  X  has  covariance  function  +  Ks, 

where  Ks  and  K N  are  known  covariance  functions  on  /  and  K s  e  H  (Ky)  ®  If  (K"N).  This 
corresponds  to  the  problem  of  detecting  a  zero-mean  Gaussian  signal  with  covariance  function 
Ks  in  the  presence  of  additive,  independent,  zero-mean  Gaussian  noise  with  covariance  func¬ 
tion  Ky.  Here  again,  the  condition  Ks  <=  H(Ky)  ®H(Ky)  is  a  regularity  condition  and  is 
related  to  the  nonsingularity  of  (4.5.1),  as  discussed  in  Chapter  2.  In  particular,  this  condition 
guarantees  that  there  exists  a  constant  0  <  C  <  such  that  Ks  c  CKN;  that  is,  Ky  dominates 
Ks  (see(l],  §U1). 

We  assume  that  Problem  (4.5.1)  is  to  be  decided  using  a  quadratic  detector,  that  is,  a 
detector  in  which  the  test  statistic  d>(X)  is  of  the  form 

<t>(X)  =  Jim  (X)  4  Jim  £  (N  )X  (q  )X  ( ),  (4.5.2) 

where,  for  each  N ,  the  infinite-dimensional  real  matrix  (<j>,-y(lV))  is  symmetric  and  has  only 
finitely  many  nonzero  elements.  The  limit  is  taken  in  the  mean-square  sense  under  both 
hypotheses;  that  is, 
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lim  E„  [(<D(X)  -  0*(X)]2}  =0,  n  =  0  and  1.  (4.5.3) 

N— **• 


Note  that  this  definition  is  very  general  and  includes  the  more  conventional  examples  of  qua¬ 
dratic  statistics  as  special  cases.  Since  K N  dominates  the  limits  in  (4.5.3)  exist  if  and 
only  if  (see  [3]  for  details) 

0=  Urn  Var0(<t>jv(X)-a>M(X)} 


=  lim 
N  M—*** 


Var0 


•  £  [<)>/> (N )  -  $ij (M )]X (r, )X(tj) 


(4.5.4) 


=  2 


4  lim 
\nm-+~ 


2  2  [<MN>  -  tojmmuW)  -  +um]KTA'kJti)Kfi(.thtj) 

i  .7=1  k.l= 1 


and 


0  =  lim 

NM-*~ 


lim 

NM-*~ 


E0{OA,(X)-Ow(X)} 

-  I'toijW-tijWMsitjJi) 

‘•y=1 


(4.5.5) 


Recall  that  the  space  H(KN)  is  equivalent  to  the  RKHS  where  the 

reproducing  kernel  K$  :  I2  x  I2  — ¥  R  is  given  by 

(?2>x2))  -  K fl(f lit 2^n(x1’X2)’  (ri»xi)  e 

If  we  define  <t>NK $  e  B(K&)  by 

ij= 1 

i.7=l 


then  it  follows  from  (4.5.4)  that 
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lim  ll» (K&)  =  °> 

N  M 

and  we  can  define  <&£$  e  H(K^)  as 

d>^(-,*)4  lim  %*£  (',*)  =  lim  f  ^:(^)ATN(-,/i)^N(*^). 

<V-*»  A/— *»  ( 

Clearly  then,  corresponding  to  any  statistic  d>(X)  of  the  form  (4.5.2),  there  exists  a  sym¬ 
metric  filter  d>  e  F(K$).  The  term  symmetric  refers  to  the  fact  that  d>  =  (Oy  )^=1,  where, 
for  each  finite  filter  <I>y,  the  matrix  (^(N ))  is  symmetric.  Conversely,  given  an  arbitrary 
symmetric  filter  O  e  F{K^ ),  there  exists  a  corresponding  statistic  d>(X)  of  the  form  (4.5.2) 
only  if  (4.5.5)  is  satisfied.  This  need  not  always  be  the  case,  but  for  the  remainder  of  this 
section,  we  will  ignore  any  possible  convergence  problems  and  simply  assume  that  (4.5.5)  is 
satisfied  for  any  filter  under  consideration.  In  any  case,  for  the  performance  criterion  con¬ 
sidered  here  (or,  for  that  matter,  most  other  common  criteria),  the  satisfaction  of  (4.5.5)  is 
unrelated  to  the  performance  characteristics  of  the  filter. 

Quadratic  detectors  for  deciding  Problem  (4.5.1)  are  often  compared  using  the  deflection 
ratio,  which  is  one  member  of  the  class  of  so-called  generalized  signal-to-noise  ratios  (see 
[17]  for  a  discussion  of  this  class  of  performance  measures).  The  deflection  ratio  for  Problem 
(4.5.1)  corresponding  to  a  test  statistic  <D(X)  of  the  form  (4.5.2)  with  associated  symmetric 
filter  <J>  e  ZF  (Xjy  )  is  given  by 
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D(d>;  (KSJ<&))& 


[h,{<I>(X)}  -  Eo{<IKX)}]2 
Var0{<I>(X)} 


r 


")2 


Jim  Z*,j<H)Ks 

r~  ij- 1 _ 


2 


i  .7=1  kj.  1 


2^,4>Ar^B(W)' 


We  extend  the  definition  of  D  to  include  arbitrary  filters  in  the  obvious  way;  that  is,  for  any 
<J>  €  F(K$),  we  define 


^s.  Ws>] 
2(0*jj.««3Wa)' 


(4.5.6) 


Comparing  with  (4.4.2),  we  see  that  the  deflection  ratio  takes  the  form  of  a  signal- to- noise 
ratio,  where  we  regard  I2  as  the  index  set,  Ks:  I2  -»  R  as  the  signal,  and  :  I2  x  I2  -»  IR 
as  the  covariance  function. 


It  follows  immediately  from  (4.5.6)  and  the  Schwarz  inequality  that  a  filter 
d>*  €  F(K^)  satisfies 

D  (O* ;  (KSJC& ))  =  sup  D  (O;  (KSJC& ))  (4.5.7) 

<J>€F(Kh) 

if  and  only  if  =  cKs  for  some  c  *  0,  and  the  maximum  possible  deflection  ratio  for 

Problem  (4.5.1)  is  given  by 


D(d>*;  (KsJtft))  =  V^PfsIla^j). 
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Since  Ks  e  H(K^)  ®  J5f (ATj^)  =  H(K$),  a  filter  O*  6  F(Ky)  satisfying  (4.5.7)  can  always 
be  found.  Further,  if  we  let  O7  represent  the  "transpose"  of  a  filter  O  e  F(K$),  defined  in 
the  obvious  way,  then  O1  e  F(K^),  and  =  cKs  if  and  only  if  O^ff^  =  cKs.  Hence,  a 
filter  O*  €  F(K$)  satisfies  (4.5.7)  only  if  0*4-  (<P*)T  does,  and  it  follows  that  a  symmetric 
filter  satisfying  (4.5.7)  always  exists. 

Now  suppose  that  the  pair  is  known  only  to  belong  to  some  set  and  let  U 

represent  the  associated  set  of  "signal-covariance"  pairs  ( sJC),s  e  H(K),  corresponding  to 
the  possible  values  of  (iST n  )-  The  set  K(U)  is  then  the  set  of  covariance  functions 
K  :  I2  x  I2  F  corresponding  to  the  possible  values  of  K^.  To  decide  the  stochastic  signal 
detection  problem  in  the  presence  of  these  uncertainties,  one  might  wish  to  use  a  quadratic 
detector  incorporating  a  symmetric  filter  Or  €  F  (K  (U ))  that  satisfies 


inf  D(Or;  (s^T))  =  sup  inf  D(<D;  (s^T)).  (4.5.8) 

(sJOeU  *eF(K(U))  (s.K)eO  y  ' 

We  refer  to  an  arbitrary  filter  Or  e  F(K(ZS))  satisfying  (4.5.8)  as  a  robust  filter  for  the 
game  (F  (K  (U)),U  X> ).  If  Or  is  also  symmetric,  we  refer  to  it  as  a  robust  symmetric  filter. 

As  usual,  to  find  a  robust  filter  we  search  for  a  saddle  point 
(Or;  Csl^l))  €  F(K(U))  x  U  satisfying 


0(0;  (sl^l))^0(0r;(jl^l))^0(0r;(s^)),  VOs  F(K(U)\  (sj()  e  U. 

(4.5.9) 

It  follows  immediately  (as  in  Lemma  4.4.3)  that  (Or;  (jl»^l))  is  a  saddle  point  for 
(F(K (U)),U  JD )  only  if  OrATl  =  csL,  c  *  0,  and 

D*(slJCOSD*(sJC),  V  (5,/Oe  U, 
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where 

D *  (s  JC)  4  ^svg(U))  D  (&’ (s  •*))  =  l/l\\s  j| h(Ky  (4.5.10) 

Further,  it  is  easy  to  see  that,  for  any  G>  e  F(K  (U)), 

D(<D;  (sJC))  ^D((0  +  4>7');  (s£)),  ET, 

which  implies  that  (Or;  a  saddle  point  for  (F  {K{U)),U  J))  only  if 

((Or  +  Or);  (slJCi))  is  also  a  saddle  point  Hence,  robust  filters  corresponding  to  saddle 
points  for  (F  (IK  (U  )),Ef  JD )  can  always  be  taken  to  be  symmetric. 

Now  suppose  that  the  noise  covariance  is  assumed  to  be  known  but  that  FC$  is 
known  only  to  belong  to  a  convex  set  S  <zH(K^).  The  set 


is  then  an  admissible  uncertainty  class  defined  on  the  index  set  I2.  Hence,  the  problem  of 
finding  robust  filters  for  (F(K$),Ef  J) )  is  identical  in  form  to  that  of  finding  robust  matched 
filters  when  the  noise  covariance  is  known  and  the  deterministic  signal  belongs  to  some  con¬ 
vex  set.  It  follows  by  an  argument  analogous  to  that  given  in  Corollary  4.3.8  that  a  robust 
symmetric  filter  for  <JF(K$),U  J))  exists  and  can  be  chosen  to  satisfy  OrAT^  =  sL,  where  sL 
is  the  unique  element  of  5  (the  closure  of  5  in  H  (AT jy ))  with  minimum  norm. 

If  /if  N  is  not  assumed  to  be  known,  the  situation  is  somewhat  more  complicated.  In  this 
case,  the  set  U  will  not  generally  be  convex,  and  we  cannot  apply  the  results  of  previous  sec¬ 
tions.  However,  if  we  let  U'  4  co(U)  (the  convex  hull  of  U),  then  it  is  easy  to  see  that  U’ 
is  an  admissible  uncertainty  class  on  the  index  set  I2.  The  problem  of  finding  robust  filters 
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for  the  game  (F(K(U')),U'J) )  thus  takes  the  form  of  the  robust  matched  filtering  problem 
discussed  in  Section  4.4,  and  we  can  apply  previous  results  to  the  search  for  saddle  points. 
Since  Vf'  =  co (17),  we  again  have,  for  all  O  e  F(K  in')), 

D(0;  (sJO)  £D((0  +  07’);  (sJO),  V  (sJC)  e  U\ 

and  it  follows  that  robust  filters  corresponding  to  saddle  points  for  {F (JK(U')),U'JD)  can  be 
taken  to  be  symmetric. 

It  is  true,  of  course,  that  a  saddle  point  for  ( F(K  (Uy),U'J> )  need  not  be  a  saddle  point 
for  the  original  problem  (F  (K{U)),U  JD ).  However,  if  (Or;  (s LJC l))  is  a  saddle  point  for 
(FOKiUy.&'J)),  then  it  follows  from  (4.5.9)  and  (4.5.10)  that 

0  <  WlsdtB(KO  =  D(*rI  Si? (Or;  (sJO),  *  (s  JO  e  U. 

Hence,  the  performance  of  the  filter  Or  (as  measured  by  the  deflection  ratio)  is  bounded 
below  by  Vi||sL|| for  all  (sJC)e  U.  In  this  respect.  Or  remains  robust  for  the  smaller 
problem  (F(K(U)),U  JJ)  although  it  is  possible  that  there  are  yet  other  filters  whose  worst- 
case  performance  over  U  strictly  exceeds  that  of  Or. 

To  illustrate  the  robust  quadratic  detection  problem,  we  consider  the  following  simple 
example  in  which  the  covariance  function  of  the  stochastic  signal  is  assumed  to  be  known. 
For  the  sake  of  convenience,  we  consider  a  set  U'  that  is  larger  than  co(U). 

Example  4.5.1:  Let  /  =  Z ,  the  set  of  integers  and  let  Q  =  [-tc,jc].  Suppose  that  the 
zero-mean,  Gaussian  signal  has  known  covariance  function  K  s  satisfying 

£j*s(i,y)]2<~. 
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Note  that  this  implies  that  the  signal  is  a  nonstationary  process.  Let  o  be  the  (necessarily 
symmetric)  two-dimensional  Fourier  transform  of  the  covariance  function  K$,  that  is, 
a  €  L2(Cl2)  and 

Ks(m,n )  =  fet<smelX*a((i>A)dGsd'K  V  (m,/i)  e  Z2. 

on 


Suppose  that  the  noise  process  is  a  zero-mean  stationary  Gaussian  process  with  covari¬ 
ance  function  ATN  having  power  spectral  density  v  that  is  known  only  to  satisfy 

-“|v(G))dci)  =  p,  (4.5.11) 

and 


V;(o>)  <;  v(co)  £  v^©),  V  o)  6  Q,  (4.5.12) 

where  vz  and  vu  are  known  functions  satisfying 

0  <  inf  vz(©)  <  sup  v„(©)  <  «>, 

(ocQ  ooeh 


and 


-^-Jv/(©)d©  <  p  <  ~Jvu(©)d 


to. 


Corresponding  to  any  power  spectral  density  v  satisfying  (4.5.11)  and  (4.5.12),  we  have 

^((m!,*!);  (m2,/i2))  =  KN(m1^n2)KN(ni’n2)  = 


and  it  follows  that  the  set  U  is  given  by 
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U  4  s  =KS  and  (m2,n2))  =  f \ei^mi~m^eiUni~nM(oM\)da)dX, 

[  4n2aa 

* 

where  v,(co)  £  v(co)  <  vu(co)  and  fv(co)da)  =  p 

2n  a 

For  this  problem  then,  it  is  convenient  to  take  the  set  U'  n  co(U)  to  be 

U'  k  s  =  Ks  and  K^m^O;  (m2,n2))  =  -^r  f  Jeico(mi_mj)e,X(ni“',l)ii(to,X)da)dX, 

[  4jr  nn 

where  v,  (co)v,  (X)  <;  n(oo,X)  £  vM(co)vB(X)  and  f  fT](co,X)d  o>  =  p2  % 

4*2CX2 

and  we  seek  a  robust  symmetric  filter  for  the  game  (JF  (K  (U  y),U ' ■£> ). 

Note  that  every  AT  €  K(JJfr)  takes  the  form  of  a  covariance  function  for  a  two- 
dimensional  stationary  random  field  with  power  spectral  density  t\  satisfying 

v^oj )v,(X)  S  T|(oo,X)  £  vM  (co)vM  (X),  v  (toA)  e  fi2  (4.5.13) 

and 

~r  f  Jr|((o,X)d(odX  =  p2.  (4.5.14) 

4tr  on 

Further,  for  every  K  €  K(U')  with  power  spectral  density  rj,  the  RKHS  H  (K)  consists  of 
functions  of  the  form  (see  [48],  §8) 

/(m,n)  =  f  Jei<nme‘^F(co,X>n(co^)dci)dX,  VC m,n)eZ 2,  (4.5.15) 

4tc2  fan 


where 


Ilf  llff(K)  ^  IF (©,X)  I ^(to^d cod X  <  oo. 


(4.5.16) 
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Also  note  that  each  K  e  K(Uf)  dominates  K(U').  In  particular,  if  K0  e  K  (U'),  then 
for  any  other  K  e  K(U'),  we  have 


K  <c 


inf  Vi  (to) 
(.men 


It  follows  from  Corollary  4.4.5  that  (Or;  (j  lJK l))  is  a  saddle  point  for  (JF{IK{U'))M'X>)  if 
and  only  if  (s^l)  is  ^east  favorable  for  (F  and  OrK" L  =  cs  L  f°r  some 

c  *  0.  Clearly,  in  this  case  we  will  have  s^  =  Ks.  Hence,  if  we  can  find  K L  e  K(Ur)  satis¬ 
fying 

D\sL^O^lWs\\a(Ko^l^s\\a{K)sD\SLJC),  v  K  e  IK  (IT), 

(4.5.17) 

then  the  filter  Or  satisfying  0RJf  L  =  Ks  will  be  a  robust  filter  for  (F  (K  ( U  ")),&' JD ). 

To  find  AT L  satisfying  (4.5.17),  we  note  that,  for  any  K  e  K(U')  with  power  spectral 
density  T|, 

Ars(m,«)  =  ^ffeiwe‘^-^^7l(co,X)dcodX,  V(m,n)€  Z2. 

4ic2aa 


Since  inf  ti(coA)  >  0,  it  follows  from  (4.5.15)  and  (4.5.16)  that  Ks  e  H (K)  and 

(<a,X)€  fi2 


(4.5.18) 


Hence,  finding  KL  6  K(U')  satisfying  (4.5.17)  is  equivalent  to  finding  the  function  qL 
minimizing  (4.5.18)  subject  to  the  constraints  (4.5.13)  and  (4.5.14).  This  problem  has  been 
solved  in  [24],  where  it  is  shown  that  r\L  satisfies 
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tjl(co,X)  =  max  {v, (£0^,(1),  min  {c  I  a(co,X)  I ,  vu  (co)vM (X)}},  V  (o>,X)  e  fi2, 
if  c  >  0  can  be  found  such  that 

J I  TU/fflW  03d  X  =  p2. 

4ir  nn 

Assuming  that  such  a  c  exists,5  we  define  the  filter  Or  as  the  infinite  dimensional  matrix 
with  entries  <j>J^  given  by 

<j>^,  ^  — —f [ei<aneiX/,^^dakiXt  *(m,n)e  Z2. 

4k2  an  til(o),X) 

Since  inf  T]L(G3,X)  >  0,  —  €  L2(Q2),  and  <I>r  is  well-defined.  Further,  since  ~  is  sym- 
(<a,X)en2  tlL 

metric,  so  is  Or.  Rnally,  for  all  (m  ,n )  €  Z2,  we  have 

0RA'L(m,/i)  =  f)  Q$Ki((mji);  ( ij )) 

=  -M  (C0,X)da)dX 

4tc266  ’11^) 

=  K  s(m  ,n ). 

It  follows  that  is  a  robust  symmetric  filter  for  the  game  (F{K  (U')),U'X> ). 

4.6.  Conclusion 

In  this  chapter,  we  have  investigated  the  application  of  reproducing  kernel  Hilbert  space 
theory  to  the  problems  of  robust  signal  detection  and  estimation.  In  particular,  we  have 
characterized  minimax  robust  solutions  for  the  general  L2-estimation  problem  in  the  presence 


sNote  that  such  a  c  always  exists  for  the  case  in  which  a(©A)  >  0  for  all  (o>X)  e  Cl2. 
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of  uncertainty  regarding  the  second-order  structure  of  the  problem.  Also,  we  have  discussed 
conditions  under  which  robust  solutions  to  this  problem  are  guaranteed  to  exist. 

These  results  for  the  L2-estimation  problem  are  remarkably  similar  to  results  given  by 
Poor  in  [53]  relating  to  robust  matched  filtering.  In  order  to  more  clearly  reveal  the  similari¬ 
ties  between  the  two  problems,  we  have  reformulated  the  robust  matched  filtering  problem  in 
an  RKHS  context.  Within  this  context,  we  have  seen  that  most  of  our  results  pertaining  to 
robust  L2  estimation  are  also  valid  for  the  robust  matched  filtering  problem.  Many  of  these 
results  can  be  seen  to  be  extensions  of  those  given  in  [53]. 

Finally,  we  have  considered  the  problem  of  robust  quadratic  detection  of  Gaussian  sig¬ 
nals  in  the  presence  of  Gaussian  noise  where  the  deflection  ratio  is  used  as  a  performance  cri¬ 
terion.  We  have  shown  that  this  problem  can  also  be  formulated  in  an  RKHS  context.  Using 
this  formulation,  we  have  shown  that  the  robust  quadratic  detection  problem  is  essentially 
analogous  to  the  robust  matched  filtering  problem.  It  should  be  mentioned  that  the  deflection 
ratio  may  not  be  the  best  measure  of  performance  for  the  robust  quadratic  detection  problem. 
While  it  is  well  known  that  the  deflection  ratio  possesses  certain  desirable  properties  in 
small-signal  situations  (see,  for  example,  [73]),  a  better  measure  of  performance  in  the  general 
case  may  be  the  so-called  modified  deflection  ratio.  A  discussion  of  this  performance  measure 
in  relation  to  the  deflection  ratio  and  other  generalized  signal-to-noise  ratios  is  given  in  appen¬ 
dix  D.  The  robust  quadratic  detection  problem  incorporating  the  modified  deflection  ratio  as  a 
performance  criterion  can  also  be  formulated  and  analyzed  in  an  RKHS  context. 

The  approach  presented  in  this  chapter,  in  addition  to  providing  a  unified  view  of  the 
problems  discussed  above,  provides  a  formulation  that  is  useful  for  investigating  robustness 
properties  in  other  problems  to  which  RKHS  theory  applies.  For  example,  in  a  recent  paper 
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[51],  Picinbono  and  Duvaut  discuss  the  design  of  optimal  linear-quadratic  detection  and  esti¬ 
mation  strategies  in  non-Gaussian  situations.  It  appears  that  this  problem  can  be  reformulated 
in  RKHS  terms  for  the  purpose  of  designing  robust  linear-quadratic  detectors.  The  structure 
and  analysis  of  the  problem  would  undoubtedly  be  very  similar  to  that  presented  in  Section 
4.5. 

Also,  many  different  approaches  to  signal  reconstruction  and  spectrum  estimation  are 
particular  examples  of  a  more  general  RKHS  formulation  (see,  for  example,  [12]  and  [79]). 
When  formulated  in  this  general  setting,  these  problems  are  seen  to  be  analogous  to  the  gen¬ 
eral  /^-estimation  problem.  This  being  the  case,  it  would  seem  natural  to  apply  the  minimax 
techniques  discussed  in  this  chapter  to  the  problem  of  signal  reconstruction  in  the  presence  of 
noisy  observations.  The  design  of  robust  signal  reconstruction  and  spectrum  estimation  pro¬ 
cedures  in  this  context  is  an  interesting  topic  for  further  investigation. 


Ill 


CHAPTER  5 
CONCLUSION 

In  this  thesis,  we  have  considered  several  different  statistical  signal  processing  problems, 
and  we  have  applied  reproducing  kernel  Hilbert  space  theory  to  the  study  of  each.  The  thesis 
is  not  intended  to  be  a  study  of  the  applications  of  RKHS  theory;  our  primary  goal  has  been 
to  investigate  the  signal  processing  problems  presented  herein.  Nevertheless,  the  work  clearly 
demonstrates  that  RKHS  techniques  can  be  very  useful  and  have  a  broad  range  of  application. 

The  thesis  is  divided  into  two  principal  parts.  In  the  first  part  (Chapter  3),  we  considered 
the  problem  of  signal  detection  in  fractional  Gaussian  noise.  We  were  able  to  answer  several 
interesting  questions  related  to  this  problem;  for  example,  we  gave  conditions  that  are  neces¬ 
sary  and  sufficient  for  the  problem  to  be  nonsingular,  we  developed  whitening  filters,  and  we 
characterized  the  optimal  detector  in  terms  of  the  likelihood  ratio.  We  have  left  unanswered, 
however,  many  equally  interesting  questions.  For  instance,  we  have  not  considered  the  prob¬ 
lem  of  sequence  detection  in  FGN,  nor  have  we  considered  what  mechanisms  might  be 
expected  to  give  rise  to  additive  FGN  on  communication  channels.  In  short,  there  is  much 
interesting  research  yet  to  be  done  with  respect  to  this  problem  and  with  respect  to  other 
aspects  of  statistical  signal  processing  in  the  presence  of  strongly  dependent  noise. 

In  second  part  of  the  thesis  (Chapter  4),  we  studied  some  problems  in  robust  detection 
and  estimation.  Applying  RKHS  techniques,  we  characterized  minimax  robust  solutions  for 
L 2  estimation,  matched  filtering,  and  quadratic  detection  in  the  presence  of  uncertainty  regard¬ 
ing  the  relevant  statistics.  We  also  gave  some  results  regarding  the  existence  of  solutions  to 
these  problems.  The  RKHS  approach  provided  a  general  and  unified  framework  in  which  to 
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analyze  these  problems,  and  we  believe  that  it  can  also  be  profitably  applied  to  other  prob¬ 
lems,  such  as  robust  signal  reconstruction. 
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APPENDIX  A 

SOME  LEMMAS  REFERENCED  IN  CHAPTER  3 


In  this  appendix,  we  state  and  prove  several  technical  lemmas  that  were  referenced  in 
Chapter  3.  Throughout  the  appendix,  jj,  will  be  the  measure  defined  by  (3.2.11),  and  AH  will 
be  the  subset  of  functions  in  L2(R )  defined  by  (3.2.12). 

Lemma  A.1:  If  /  €  L X(R )  n  L2{R),  then  /  e  AH. 

Proof:  Let  /  €  Ll(IR)  n  L2{R ).  We  must  show  that  /  e  L2(p).  To  this  end,  recall 
that,  since  /  e  Ll(R)  n  L2(R),  it  follows  that  /  e  L2{R ),  and,  for  all  0)  e  R , 


l/(0»l  = 


Je-iw,/(r)dr 


S  j \f{t)\dt 


Hence, 


~  |  l/(co)l2l(o!1-2Wdco  £  l/(o))l2lcol1-2Wdo)  +  ~  J  l/(co)l2dco 

\  j  “ 

£  ~  J  |«,|i-2»d«  +  ^-  J  I  /  (co)  1 2d  co 


< 


Therefore,  /  €  L2(p)  and  /  e  Aw.  ■ 
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Lemma  A 2:  Let/,#  e  Ll(R)  n  L2{R).  Then 

••  _  om  om 

~  jf(,-<0)g(-<0)  \co\x.-v1d(o  =  VhH(2H-1)  j  jf(t)g(s)  I t-s  \w~2dsdt,  (A.l) 
where  VH  is  defined  by  (3.2.3). 

Proof:  Suppose  first  that  /  and  g  have  compact  support.  Then  it  follows  straightfor¬ 
wardly  from  Young’s  inequality  (see,  [15],  page  232)  that 

M  <M 

j  J  1/(0 1  l*(j)l  I  t-s  I  w~2dsdt  <o°. 


Hence, 


;V 


J  /  (-©)  g(-oa)  1  co  1 1~2// d co  lim  -  [/(-G))  #(-to)  I co 1 1  wdto 

2k  jL  N-*o,  2k  Jy 


/  J/(0*CO 


=  J  J7(0$C») 


l  N 

-L  f ei<^t~s)\(a\l~2H d 
2  kJn 


£0 


dsdt 


N 


~Jcos(o)lr-s  I )co1_2// da 


dsdt 


=  Jim  j  jf(t)g(s)\t-s\2H~2 


N  U-i  I 


—  f  cos  (X)X1-2HdX. 
7t  n 


Now, 


X 

lim  —  [cos(K)Xl~v1  dX  =  VhH(2H-1), 

'x— K  n 


and  there  exists  a  constant  K  >  0  such  that,  for  all  x  >  0, 
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Jcos(X)X1_2WdX 


Therefore,  by  the  dominated  convergence  theorem, 


<  K. 


N 


lim  j  jf(t)g(s )  I r— s  I 


1H-2 


N\t-s\ 


—  f  cos  (X)X1-2^dX 

k  A 


dsdt 


=  VhH(2H-\)\  jf(t)g(s)\t-s\w-2dsdt, 

and  (A.l)  follows  for  /  and  g  with  compact  support.  To  prove  (A.l)  for  general 
f,g  €  L\R)rx  L\R ),  let  [fN  and  [gN  }*T=1  be  defined  by 


fs  (f  )  =  /  (*  V jV](f ). 


t  e  J?, 


and 


gN(t)=g(t)I[-NJV](t),  t  €  R. 

Clearly,  fN  — >  /  and  gyv  ->  g  in  Ll(R)  and  L2(R ),  and  it  fellows  from  the  proof  of  Lemma 
3.2.1  that  fN->f  and  gN  ->g  in  L2(|i).  Since  fN  and  gN  have  compact  support  for  all  N ,  we 

get 

oo  oo  _ 

~  f / (—co) g (-<o)  lcoll~2//dco=  lim  [/^(-co) g^(-<o)  \(o\l~2Hd(o 


2it 


=  lim  VfjH (2H-1)  f  J  fN(t)gN(s)  \t-slw~2dsdt 

N—*— 


N  N 


=  \imVHH(2H-\)  j  \f(t)g(s)\t-s\2H~2dsdt 
N  Kt  A I 


-N-N 


=  VhH(2H-1)  j  jf(t)g(s )  It-s  \w~2dsdt. 
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_10X 

Lemma  AJ:  Let  S  4  { — : - ;  t  e  R  }.  Then  5  spans  L2(li). 

103 

Proof:  Clearly,  S  A  {/[o,/]? 1  €  R }  spans  L2(JR),  and,  since 


-  /  [0^]( — C0)»  ^  C 06  J?, 


it  follows  from  the  Fourier-Plancherel  theorem  that  5  spans  L2(R ).  Hence, 


{&(«)  k 


.  e,ay-l  w  ,,  «sgn(a>Xtf-V4)—  , 

i  — : - lcol"_/1e  L ;  r  €  R  }  spans  I/(|x)  and  it  is  sufficient  to  show  that 


any  g,  can  be  approximated  in  L2(ji)  by  a  finite  linear  combination  of  functions  in  S.  Now, 
it  is  straightforward  to  show  that  g,  e  L2{R)  with  inverse  Fourier  transform 
g,  e  Ll(J?)  n  L\R)  given  by 


g,(z)=rvhi)  [w^X'wl**  -  ltil4-")  +  /(_,o1«lr«l'4-"], 


X  G  R. 


Since  g,  e  Ll(R)  n  L2(R ),  it  is  clear  that  there  exists  a  sequence  of  functions 

(Mx)  =  2 / [0^,- ] ( — x) ) /H=  l  converging  to  g,  in  Ll(R)  and  L2(R).  It  follows  from  the  proof 
«=i 

of  Lemma  3.2.1  that  <j>„  — »  g,  in  L2(\i),  and,  since 


<M«)  = 


5.  eia*'-l 


v  c o  e  R, 


the  result  follows. 


Lemma  A.4:  Let  [0,7]  be  a  compact  interval  and,  for  t  e  [0,7],  let  f,  be  given  by 


l 

/,0O  A  to*' J(M )M  u  e  t°’r^ 


Then  the  set  of  functions  [ft  ;  t  e  [0,7]}  spans  L2([0,7]). 
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Proof:  It  suffices  to  show  that  the  only  function  in  L2([0,T])  orthogonal  to 
[0,7'] }  is  the  zero  function.  To  this  end,  let  g  e  L2([0,7])  and  suppose  that,  for  all 
t  €  [0,T], 


0  =  jf,(u)g(u)du 
o 


g(u)du 


- 1 -  fT 

nH-W)  J 


j(x-u  i1  ~^u  V4“w  g(u)du 


d  x. 


Differentiating  with  respect  to  f ,  we  see  that,  for  almost  all  t  e  [0,T], 


0  =  tv  u  "  (*  ~u  Jf(u> du- 

1  \n—/2)  q 


Hence,  for  all  t  e  [0 ,T], 


0  =  — — -  f(r -5  - 


1 


J(s -u )H ~^u g(u)  du 


ds 


—  — — — J— — —  ■  fy 'Ar-H  q(u) 
r(3/2-H)r(H-'A)  l  * 


jit-s)'^  (s  -u'f1  iflds 


du 


=  g(u)du. 


Differentiating  again  with  respect  to  t,  we  see  that,  for  almost  all  t  e  [0,T], 


0  =  t'^H  g(t). 
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Hence,  g  =0  almost  everywhere  in  [0,7],  and  it  follows  that  {/,;  /  e  [0,7])  spans 
L2([0,7]).  ■ 
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APPENDIX  B 

PROOF  OF  THEOREM  4 3.S 

In  this  appendix,  we  prove  Theorem  4.3.5.  Throughout  the  appendix,  /  will  represent  an 
arbitrary  index  set,  ZD  an  admissible  uncertainty  class  defined  on  /,  IK  {ZD)  the  class  of  covari¬ 
ance  functions  contained  in  ZD ,  and  F  {K {ZD))  the  class  of  filters  defined  on  K {ZD).  In  order 
to  prove  Theorem  4.3.5,  we  will  need  the  following  technical  lemmas. 

Lemma  B.l:  For  any  covariance  function  K  defined  on  /  and  any  C  >  0, 
B{CK)  =  B{K),  and 

/.  _ L  /. 

\**/®(CK)-  £T\  »*/»(*)• 

Further,  if  K0  and  Kx  are  two  covariance  functions  on  /,  and  K\  <c  CK0,  then 
H{K{)  c  B{K0),  and 

Vllow  *  Cfflijtj.  V /.»(*,). 

Proof:  See  [1],  §1.7  and  §1.13.  ■ 

Lemma  B.2:  If  K0  and  Kx  are  covariance  functions  on  /  and  K  x  <  CK0,  then 
h  €  F{Kq)  implies  h  e  F{KX). 

Proof:  Let  h  =  {hN}jv=i>  where  hj,  h2,  •  •  •  are  finite  filters.  Then,  since  CKQ-K{  is 
nonnegative  definite  on  /  and  h  s  F  {K0), 
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0  ^  m  vm  Ki-hNK illir^,) 

=  ju  wm  1*  (**W  “fyv  )/ 

^  w  vm  ^  (AH***  (**Ar“tyv)) 


=  lim 

M 

=  0. 


It  follows  that  h  e  F^).  ■ 

Lemma  B.3:  Suppose  KxcCKq  and  let  Ka  =  (l-a)AT0  +  aJCl  for  0  <  a  <  1.  Then 
B  (K  a)  =  H  (Kq),  and,  for  all  a  sufficiendy  small, 

(l-a)/if0  <  ATa  «  (l-a+Va)K 0.  (B.l) 

Hence,  by  Lemma  B.l, 


1-a+Va  S  ^  (B.2) 

and  II'IIm (k„)  “>  IIH»(Ar0)  as  a  0. 

Proof:  Qearly,  (l-a)A'o  <*Ka<c  (C+\)K0.  This  establishes  the  left-hand  side  of  (B.l) 
and  implies  (by  Lemma  B.l)  that  H(KJ  =  B(K0)  for  all  0  <  a  <  1.  Also, 


(l-a+Va)tf0  -  Ka  =  (l-cfrWoflffo  -  (1-a )K0  -  a Kx 
=  VaK0  -  aX  2 
1 

Va' 


=  a 


^o-^i 


Hence,  for  all  0  <  a  <  C"2,  Ka  <  (l-a+Va)X0,  which  establishes  the  right-hand  side  of  (B.l) 
and  proves  the  lemma.  ■ 
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Lemma  B.4:  Let  Kq  and  be  covariance  functions  such  that  K\  <*  CKQ.  Let 
sQe  H(K0)  and  Ka  =  (l-a)AT0  +  cxAT x  for  0  <  a  <  1.  Choose  h0eF(AT0)  such  that 
hoATo  =  s0  and  ha  6  F{K^)  such  that  haATa  =  s0  (recall  H (K0)  =  H (Ka),  by  Lemma  B.3). 
Then 

lim  ((ha-h0)AT 0,  (ha-h0))  =  Urn  ||haAT0  -  h0/C0||^(^a)  =  0,  (B.3) 

a— >0  a— »0 

and 

lim  ((ha-h0)Af ,,  (h^-ho))  =  lim  ||haA: j  =  0.  (B.4) 

a— +0  a—*0 

Proof:  Notice  first  that  (by  Lemmas  B.l  and  B.3)  H(K{)  Q  U  (K0)  =  H  (Ka).  Let 
h0={hJ5}^1.  Then 

0  £  lim  ||f0  ” 

N  — *»• 

S  Jim  -^\\s0  -  h^oll^(Aro)  (B-2» 

-  ~  h0^ollff(Ara) 

=  0. 

So  ligATo  converges  to  Sq  in  H(K^}  as  well  as  in  H(K0).  Further,  by  Lemma  B.2, 
F  {K  a)  =  F  (K 0)  c  F  (K  i).  and  it  follows  from  an  argument  similar  to  the  above  that  h^  { 
and  haAf  t  both  converge  in  H(K 0)  and  H(Ka)  as  well  as  in  H  (K  j).  Hence, 
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((ha-h0)*0.  (ha-h0))  <S  ((ha-h0)*a,  (ha-h0)) 

-  -j~l l|ha*a  ~ 

=  “  (l~a)hO^O  ”  “ho^lllfl^,,) 

=  *j~l Ifco  ~  (l-a>fo  ~  othoATill bvcj 

-  -^lUo  “  illir^,)- 

(B.3)  follows  by  letting  a  -»  0  and  noting  that  Hs0  -  h0^1||a(jfa)  remains  bounded  (by  lemma 
B.3).  Finally,  since  AT j  c  CK0, 

0  £  '(ha— ho)*!,  (ho-h0))  <;  C  ((ha-h0)*0,  (ha-h0)), 

so  (B.4)  follows  from  (B.3).  ■ 

Proof  of  Theorem  4.3 .5:  It  follows  immediately  from  lemma  B.2  that  hR  e  IF  (IK  ( U)). 
By  Lemma  4.3.2,  M* (s  JK)  =  o|  -($,$)*(*>  for  (s,*)  e  U,  and  by  Lemma  4.3.3,  M*  is 
concave  on  U.  Therefore, 

\s  L’  5  UB(.KO  ^  \s »  s)b {K)>  V  (s  e  Cr> 

if  and  only  if,  for  all  (sJC)  e  U, 

~  [(•*  a*  sa/B(Ka)  ~\SL>  5  Off  (/tru)  (  -  °> 
a—*0  uv.  J 

where  sa  =  (l-<x).sL  +  and  Ka  =  (1  -a)*L  +  <xK.  Separating  tenns  and  applying  Lemma 
B.3,  we  get 
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ton  “  ^a»  saiB(.Ka)  “  (^L*  =  2  \S>SL )b(KO  “  2  Xs L>  sL/B(Kl) 

+  tiro  ~  ^sL,  s^n(Ka)  -  [sL,  5i)/r(ArL)j 
=  2  (s ,  hR)  -  2  (s  L,  hj \)a(Ko 

+  tiro  •—  |s L>  5 L/ll (AT a)  “  Vs L>  ^Uir (ATjj- 

Now,  as  in  the  proof  of  Lemma  B.4,  we  find  ha  6  F(JK(U))  such  that  haATa  =  wifh  con¬ 
vergence  in  both  HiKfj)  and  Then 

\s  L*  sUB(KO  =  \ha*a>  ha^a/H(Arj 

=  (1-a)2  (hatf  L,  he,)  +  2a(l-a)  (h aK,  ha)  +  a2||ha/ni^(^iJ, 


and 


\SL>  sCb(KJ  =  (tia^a*  ^a/ 

=  (1-a)  (h^L,  ha)  +  a  (h aK,  h«). 

Hence, 

tiro  ~  |s l>  5 l)» (ATJ  “  L>  s L/B(Ko  j  =  tiro  j(l— <x)  (haAT L,  ha)  +  (2a-l)  (haAT ,  ha) 

-  a  IlhatfU^jj 

=  (hRtf L,  hR)  -  (hR/C ,  hR)  (by  Lemma  B.4). 

=  (*L>  hR)-(hRtf,hR). 


Therefore, 


lim  —  \ls 
a-*0  a  0 


o»  sa/B(Ka)  -\SL’S  UB(KO 


~  2  Vs  *  tify  \$L>  hR/ 


-  (hRAT ,  h 


\ 

R /• 


It  follows  that  (4.3.9)  holds  if  and  only  if  (4.3.6)  holds.  This  proves  the  theorem.  ■ 
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APPENDIX  C 

RKHS  APPROXIMATION  LEMMA  AND  PROOF  OF  LEMMA  4.4.8 


RKHS  Approximation  Lemma:  Let  /  be  a  separable  metric  space  and  let  [IN  )fj=x  be  a 

ao 

sequence  of  subsets  of  /  that  is  monotone  increasing  (/[C/jC  •  •  •  )  and  such  that  is 

dense  in  /.  Let  K  be  a  continuous  covariance  function  on  7,  and  let  KN  be  the  restriction  of 
K  lo  1^.  Let  s  be  a  continuous  function  on  /  and  let  Sfj  be  its  restriction  to  IN.  If 
s  €  H{K),  then  sN  e  H(KN)  for  all  N  and 

IIs  II* (K)  ~  jjira  11% IIji ucNy 

Conversely,  if  %  e  H(KN)  for  all  N.  then 

IMW<*w)  <  00 


Choose  0  <  a  <  1  and  let  sa  =  (l-a)s  +  (xrL  and  K,,  =  (l-a)K  +  aKL.  It  follows  (see  the 
proof  of  Lemma  4.4. 1  or  [53],  Property  2)  that 
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A(hR;  (sa,KJ)  Z  (1  -a)A(hR;  (s, K»  +  aA (hR;  (sl,Kl)) 

<  A(/jr;  (5l,Kl)). 

Now,  since  S  c  H0  is  convex  with  a  nonempty  interior,  it  is  easily  established  that  the  inte¬ 
rior  points  of  5  are  dense  in  S.  Further,  since  Kq  is  stricdy  positive,  the  range  of  Ka  is 
dense  in  H0.  These  facts  imply  that  there  exists  a  sequence  {s„  }“=1  c  S  n  H (K^  such  that 
1  \sn  -  sa\\Jio  0.  It  follows  that  (/zR,  sn)Ba  [hR,  s^Bo,  and,  since  (hR,  KahR)„o  >  0, 
A(/ir;  (sn  .K,,))  ->  A(/ir;  (.s^K^)).  Hence,  there  exists  an  sN  such  that  (%,Ka)  e  U'  and 

A(h R;  (sN,KJ)  <  A(h R;  (s l,Kl)). 

This  contradicts  the  hypothesis  that  (hR,  (s  l.Kl))  is  a  saddle  point  for  (IF  (IK  (U')),U',&).  ■ 
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APPENDIX  D 

GENERALIZED  SIGNAL-TO-NOISE  RATIOS  IN  QUADRATIC  DETECTION 


D.l.  Introduction 

In  this  appendix,  we  discuss  some  properties  of  the  so-called  generalized  signal-to-noise 
ratios  with  regard  to  their  use  in  evaluating  the  performance  of  quadratic  detectors  used  to 
discriminate  between  two  Gaussian  hypotheses.  A  somewhat  more  detailed  discussion  is 
given  in  [6].  Throughout  the  appendix,  X  4  {X  (r),  tel)  will  represent  an  observed  process 
defined  on  some  index  set  /,  which  is  assumed  to  be  a  separable  metric  space.  We  assume 
that  X  is  a  zero-mean,  Gaussian  process  with  one  of  two  continuous  covariance  functions;  the 
object  being  to  test  the  hypotheses: 

Hq:  X  has  covariance  function  Kq 

versus  (D.1.1) 

Hjt  X  has  covariance  function  K  j, 

A  special  case  of  considerable  practical  importance  is  the  signal  detection  problem,  in  which 
Hj  corresponds  to  a  Gaussian  signal  in  additive,  independent,  Gaussian  noise,  and  H0 
corresponds  to  noise  only.  In  keeping  with  this  example,  we  define 

Ks^Ki-KQ, 


however,  one  should  keep  in  mind  that,  in  general,  K  s  need  not  be  a  covariance  function.  We 
assume  also  that  Problem  (D.1.1)  is  nonsingular. 
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It  is  common  practice  to  decide  Problem  (D.1.1)  by  using  a  quadratic  detector,  that  is,  a 
detector  in  which  the  test  statistic  <I>(X)  is  a  quadratic  functional  of  the  observed  process. 
Indeed,  the  optimal  detectors6  generally  take  this  form.  Because  it  is  difficult  to  evaluate  the 
probabilities  of  error  for  such  a  detector,  other  measures  of  performance  are  often  used. 
Among  these  are  the  generalized  signal-to- noise  ratios  (GSNR’s),  which  we  will  represent  as 
a  parametric  family  [Da,  0  £  a  £  1}.  Given  any  quadratic  statistic  <I>(X),  Da(d>)  is  defined 
as  [17] 

-E0{<t>(%))}2 

n  r<t>\  A - i - 1 - 

(l-a)Var0{O(X)}  +  aVar^X)}  ’ 

where  the  subscripts  0  and  1  indicate  expectations  taken  under  H0  and  Hj,  respectively.  D0  is 
referred  to  as  the  deflection  ratio,  which  we  discussed  in  Section  4.5,  and  D  j  as  the  comple¬ 
mentary  deflection  ratio.  We  will  show  that  every  performance  measure  in  this  class  can  be 
represented  as  a  ratio  of  inner  products  in  a  reproducing  kernel  Hilbert  space.  Such  a 
representation  has  several  advantages.  Apart  from  being  mathematically  appealing,  it  leads 
straightforwardly  to  a  characterization  of  the  quadratic  statistic  O^CX)  that  maximizes  Da(d>) 
for  each  a  e  [0,1].  Also,  it  clearly  reveals  one  of  the  weaknesses  of  the  GSNR  as  a  measure 
of  performance  for  quadratic  detectors  and  leads  naturally  to  consideration  of  an  alternative 
measure,  which  is  superior  to  the  GSNR  in  some  respects. 


‘Throughout  this  appendix,  the  term  "optimal  detector"  refers  to  a  detector  in  which  the  test  statistic  <t>(X) 
is  (with  probability  one)  a  monotone  function  of  the  likelihood  ratio  L(X). 
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D.2.  GSNR  Representation 

As  in  Section  4.5,  we  define  a  quadratic  test  statistic  as  any  random  variable  <J>(X)  that 
can  be  written  in  the  form 

O(X)  =  lim  0*(X)  4  lim  £  <\>ij(N)X(ti)X(tJ),  (D.2.1) 

N~*~  i.j= l 

where  c/  is  a  sequence  of  observation  points  ( depending  on  O(X)),  and,  for  each 

positive  integer  N,  the  infinite-dimensional,  real  matrix  (4>iy  (N ))  is  symmetric  and  has  only 
finitely  many  nonzero  elements.  The  limit  is  taken  in  the  mean-square  sense;  i.e.,  we  assume 
that,  for  n  =0  and  1, 

Jim  E„  WX)  -  «>„  (X))2  =  0.  (0.2.2) 

Now,  let  a  €  [0,1]  and  define  Ka:  I1  x  I2  -»  J?  by 

(t2.'c2))  =  (l-<x)X’o(ti,t2)^o('ci*'c2)  +  vKl(ti,t2)Kl(xl,x2). 

Clearly,  each  Ka  is  symmetric  and  nonnegativc  definite  on  I2  x  I2.  Hence,  for  each 
a  e  [0,1],  there  exists  an  RKHS  H(K ^  with  reproducing  kernel  Ka.  If  we  let 
K  =  (Xa;  0  £  a  £  1],  then  it  follows  from  (D.2.1)  and  (D.2.2)  that,  corresponding  to  each 
quadratic  statistic  O(X),  there  exists  a  symmetric  filter  Os  F  (IK),  and  (see  [3]  for  details) 
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Dam  g 


[e!{<D(X)}  -Eo(<1>(X)}]2 
(l-<x)Var0{O(X)}  +  aVar^X)} 


Jim  ZkjWK&jA) 


\N 


ij= 1 


lim  £  £  4>(y (N )§ki (N )[( 1  -ct.)K 0(tk ,r, )K 0(q ,f, )  +  oK^t^K^j)] 


IN 


i,j=  1  k,l= 1 


lim  2  $ij(N)Ks(tj,ti) 


N 


'.y=i 


(D.2.3) 


lim  2  I  (*iJjy> 


N 


i,j-\  k.l-\ 


|*rs,  ^XTovvacj] 

2  wa;„(fJ  ■ 


Note  that  in  order  to  express  Da  in  the  final  form  given  in  Equation  (D.2.3),  we  have  used 
the  fact  that  Kse  HiK^,  which  follows  from  the  nonsingularity  conditions  for  Problem 
(D.  1.1),  as  stated  in  Theorem  2.1.4. 

One  of  the  obvious  advantages  of  representing  Da  in  the  form  given  by  (D.2.3)  is  that  it 
leads  immediately  to  a  characterization  of  the  quadratic  statistic  O^(X)  that  maximizes  D  a(d>) 
for  any  given  a.  It  appears  that  such  a  characterization  was  not  previously  known  [67],  To 
characterize  <D^(X),  notice  that  (D.2.3)  and  the  Schwarz  inequality  together  imply  that 

D.m  s  Vi  (Kg,  ATsWj. 

with  equality  if  and  only  if  <MT0  =  cKs  for  some  c  *  0.  That  is. 
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*«<X)  =  Urn  ^(X) 

4  Jim  £  tij(N)X(t*)X(t}), 

maximizes  D  a(<t>)  if  and  only  if  there  exists  c  *  0,  such  that 


(D.2.4) 


cKs(tj)  =  0*X  a(r,r) 

=  Jim  0^a(M) 

w-*-  (D.2.5) 

=  lim  £  4>i*(iV)tfa((M);  (r,V/)), 

«.;=1 

where  the  limit  in  (D.2.5)  is  taken  in  H  (Kg)  (which  implies  pointwise  convergence,  as  well). 
Note  that,  even  though  we  can  always  find  <!>„  4  [4>n  )v=i  satisfying  (D.2.5),  the  correspond¬ 
ing  {<fyv(X)}jv=1  may  not  converge  in  mean-square,  so  we  cannot  guarantee  that  O^(X)  exists. 
However,  we  do  have 


D 


* 

a 


=  =  l/i  \K S’  K sin (ATo)- 


As  a  concrete  example,  let  us  consider  the  following  simple  problem. 

Example  D.2.1:  Let  /  =  {1,  2,  •  •  • ,  N],  and  assume  that  K0  and  Ky  are  positive 
definite  N  x  N  matrices.  We  are  looking  for  a  IV  x  N  symmetric  matrix  such  that  the 
quadratic  form 

<(X)  4  XrO*X 

maximizes  the  GSNR 

[Tr  OK's]2 

n  (<D)  - - - - - - 

a  2[(l-a)(Tr  K0®K0<&)  +  a(Tr  /fjO/sTjO)] 

for  fixed  as  [0,1].  It  follows  from  (D.2.5)  that  must  satisfy 
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cKs  =  (l-cOAToOX  +  oAT  !<&**! 


(D.2.6) 


for  some  c  *  0.  One  solution  to  (D.2.6),  for  c  =  1,  is  given  by 

<t>*  =  K^lKs[a-a)K^^K0  +  a^r1 

=  [(l-a^oAfr^o  +  aATJ'1  -  K XXK 0[( l-a)AT qAT  f 1  AT  0  +  aA :{fl. 


This  solution  was  established  by  diagonalizing  Af0  and  Kx  simultaneously,  but  it  is  easy 
to  verify  directly  that  it  satisfies  Equation  (D.2.6).  For  a  *  0,  we  use  a  matrix  inversion 
lemma  ([30],  page  19)  to  write 

P  k  [(1— ct)Af0ATf1Ar0  +  OtATj]-1 

=  1K -1  _  J-d^Arf^oKl-aJAroATf^o  +  oK^KqK^ 
a  a 

=  i-AT,-1  -  -i-d-a^r'sr^^r1- 

IX  IX 

Then,  for  d>  *  defined  by  Equation  (D.2.7),  we  have 

(l-a)A:o<l>aA:o  +  aK\<Kx  =  (l-cOA^F  -  KxxKqP)Kq  +  aXx(P  -  K{lK0P)Kl 

=  (l-a)ATo^Ar0  -  (l-a.)K qK xlK qPK Q  +  aAf XPK x  -  olKqPK x 
=  (\-a)KoPK0  -  (l-aWoKt'KoPKo  +  Kx  -  {\-o.)KqPKq  -  K0 
+  (l-aJAToArf^oPATo 


as  claimed.  Verification  for  a  =  0  is  straightforward. 

For  this  example,  it  is  easy  to  check  that 

D'a  =Da(Oa)  =  1/2TrO*ATs. 


132 


The  foregoing  discussion  reveals  one  of  the  weaknesses  of  the  GSNR,  to  wit:  under 
mild  regularity  conditions  on  Problem  (D.1.1),  there  exists  an  optimal  quadratic  detector 
incorporating  a  test  statistic  (X),  which  does  not  generally  maximize  Da(<t>)  for  any 
a  e  [0,1].  To  be  more  specific,  in  [23],  Kailath  and  Weinert  have  shown  that  if  Problem 
(D.1.1)  is  strongly  nonsingular1  (as  defined  in  [23]),  then  there  exists  an  optimal  quadratic 
detector  for  (D.1.1),  and  ®opt  (X)  satisfies 

<Dopr(X)=lim  £  $w'(N)X(t?P‘)X(t?p'),  (D.2.8) 

N~*mm  i,js  1 

where 

Ks(tj)  =  lim  £  (D.2.9) 

i.j-l 

However,  it  follows  firom  (D.2.5)  that  <t>op,  (X)  maximizes  Da  if  and  only  if  there  exists  c  *■  0 
such  that 

cKs(tj)  =  lim  £  <)>if'W)tfa((r,j);  (rf^r/*)).  (D.2.10) 

N-**  ij- 1 

As  a  general  rule,  (D.2.9)  and  (D.2.10)  will  not  be  satisfied  simultaneously,  and  we  will  be 
able  to  find  a  nonoptimal  quadratic  statistic  O(X)  such  that 

Da(<S>)  >  Dal<t>opt). 

For  example,  consider  the  following  simple  signal  detection  problem. 

’Actually,  by  changing  our  definition  of  quadratic  statistic  slightly  to  avoid  convergence  problems,  we 
could  drop  this  restriction.  The  interested  reader  is  referred  to  [23]  or  [57]  for  details.  In  any  case,  this  is  only 
an  issue  when  /  is  an  infinite  set  since  (D.1.1)  is  always  strongly  nonsingular  (trivially)  if  /  is  finite. 
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Example  D.2.2:  Let  /  =  { 1,  2,  •  •  •  ,  N },  X0  =  XN,  and  =  XN  +  Xs,  where  X^  and 
Xs  are  both  N  xN  covariance  matrices.  Assuming  X N  is  positive  definite,  there  exists  an 
optimal  quadratic  detector  with 

<V(X)  4  X^CX), 

where  the  matrix  is  given  by 

«V  d  KS'KsK?  =KS'  -K{'. 

In  order  for  <bopl  (X)  to  maximize  D  a,  we  must  be  able  to  find  c  *  0  such  that 

cX  §  =  {\-<x)K  Q®optK  q  +  a X  i®optK  i 

=  (l-a)X0(Xo1  -Xf^o  +  aXrfXo1  -K?)KU 

which  is  possible  if  and  only  if 

aCX^Xg)3  +  (Ja-cXX^Xg)2  +  (l-OX^Xs  =  0.  (D.2.1 1) 

In  particular,  if  the  noise  is  white,  so  that  Xj^Xg  =  ^s>  then  (D.2.1 1)  is  satisfied  if  and  only 
if  all  of  the  nonzero  eigenvalues  of  X  s  satisfy 

aX2  +  (2a-c  )X  +  (1-c )  =  0. 

For  this  example,  we  already  know  that  defined  by  Equation  (D.2.7)  satisfies  Equa¬ 
tion  (D.2.6)  and  maximizes  Da.  Hence,  if  (D.2.1 1)  is  not  satisfied,  it  follows  that  is 
nonoptimal  (<D*  *  c<$opt  for  any  c  *  0),  and  yet  Da(<D*)  >  Da(Q>opt). 

D.3.  Alternative  Performance  Measure 

Since  it  seems  reasonable  to  want  a  performance  measure  for  quadratic  detectors  that  is 
always  maximized  by  an  optimal  quadratic  detector,  one  is  led  to  search  for  alternatives  to  the 
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GSNR.  In  light  of  Equations  (D.2.8)  and  (D.2.9)  vis-k-vis  Equations  (D.2.3)  through  (D.2.5) 
a  natural  choice  is 


|n^oi,  ^sV(a:o)  a  »(*:,)  J 

2  ®K01/M(K<d  »*(«■,)  ’ 


(D.3.1) 


where  K0i:  I2  x  I2  —>  IR  is  defined  by 


*oi((fi,Ti);  (t 


The  fact  that  D^(O)  is  well  defined  for  any  quadratic  statistic  O(X)  (i.e., 
Ks  6  H(K0l)  =  H{Kq)  ®  H{Kx)  and  O  e  F(K oi))  follows  easily  from  the  mean- square 
convergence  in  (D.2.2)  and  the  nonsingularity  conditions  given  in  Theorem  2.1.4. 

Equation  (D.3.1)  and  the  Schwarz  inequality  together  imply  that 

0^(4*)  £  xh  (ATS,  Kslawd&HdCi)’ 

with  equality  if  and  only  if  <WK01  =  cKs  for  some  c  *  0.  It  follows  that  O^(X)  maximizes 
D^(O)  if  and  only  if  there  exists  c  *0  such  that 

cKs(t,s)  =4>*KQl(tj) 

4  lim  Ztij(N)K0(t,t*)K}(s,tJ). 

N~*~  i  .7=1 

As  with  it  may  be  that  no  such  exists,  but  we  always  have 

D*n  -  ^ 6s^oi)  DvW  =  1/2  <*s.  Ks)a(K0)9B(Kly 


Now,  if  Problem  (D.1.1)  is  strongly  nonsingular,  then  we  know  that  there  exists  an  optimal 
quadratic  statistic  4>opl  (X)  satisfying  Equations  (D.2.8)  and  (D.2.9).  Since  (D.2.9)  clearly 
implies  that  4>optK0 1  =  Ks  in  H(K0)  ®  U  {Kx),  it  follows  that  4>opt  (X)  maximizes  D^(O). 
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Conversely,  it  follows  fairly  easily  from  the  results  in  [23]  and  [57]  that,  even  if  (D.  1.1)  is  not 
strongly  nonsingular,  a  quadratic  statistic  <D(X)  maximizes  D^(O)  only  if  the  corresponding 
detector  is  optimal. 

For  the  signal  detection  problem,  it  is  easy  to  verify  that  can  be  rewritten  as 

-E0mX))}2 

DJd»  - - - - 1 - - - r  ,  (D.3.3) 

V*Var0{<D(X)}  +  xh  (Var^OfX)}  -  Vars{<I>(X)}  J 

where  Vars{d>(X)}  is  the  variance  of  <D(X)  when  only  the  signal  is  present.  In  this  form,  it  is 
clear  that  is  equivalent  to  the  so-called  modified  deflection  ratio  (see  [17]).  The  denomi¬ 
nator  in  (D.3.3)  can  be  interpreted  as  the  average  of  the  variance  of  O(X)  in  the  presence  of 
noise  only  and  the  variance  of  <D(X)  due  to  noise  in  the  presence  of  the  signal.  Viewed  in 
this  fashion,  becomes  somewhat  more  intuitively  appealing. 

As  a  final  remark,  we  note  that,  while  it  is  not  directly  related  to  detector  error  probabil¬ 
ity  in  any  obvious  way,  may  be,  in  general,  a  better  predictor  of  detector  performance 
than  the  GSNR.  In  particular,  for  the  case  of  slow  and  fast-fading  channels  considered  by 
Gardner  in  [16],  Dfi  displays  significantly  less  anomalous  behavior  than  the  GSNR  when 
viewed  as  an  indicator  of  performance  gain  with  increasing  observation  time.  In  general,  this 
conjecture  is  based  on  the  fact  that  D^(d>)  is  essentially  a  measure  of  the  distance,  in 
H(K0)  ®  H  (K^),  between  OAT01  (suitably  normalized)  and  the  nearest  constant  multiple  of 
Ks.  To  be  more  precise, 
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d2mm,  Ks)  A  Jnt  ll-j^jj-  "  slf 


=  1  - 


IJATsIl 


2  D*m 


where  ||  •  ||  is  the  norm  in  H(Kq)  ®  H  ( K\ ).  Hence,  if  Oj  and  d>2  are  two  quadratic  statistics 
and 


D  n^i)  >  £>h(<&2)» 


then 


d\^K oi,  Ks)  <  d2(<J>2K0l,  K s). 

Since  the  optimal  test  statistic  is  essentially  defined  in  H(Kq)®H(K{)  and  corresponds  to 
Ks,  the  detector  incorporating  <X>j  can  be  regarded  as  being  "closer"  to  the  optimal  detector 
than  the  detector  incorporating  <J>2.  Heuristically  then,  can  be  regarded  as  a  measure  of 
the  deviation  of  a  quadratic  detector  from  the  optimal.  On  this  basis,  one  might  expect  D  ^  to 
be  a  fair  indicator  of  detector  performance.  (See  [11]  for  a  discussion  of  a  similar  property  of 
the  efficacy  of  nonlinear  detectors  for  deterministic  signals.) 
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